I want to observe the progress of a file upload. ...

# ktor

[Rok Oblak]

I want to observe the progress of a file upload.

My client appends a byte array to the form data part:

setBody(MultiPartFormDataContent(formData {
    ...
    append("file", bytes, Headers.build {
        append(HttpHeaders.ContentType, ContentType.Application.OctetStream.toString())
    })
}))

but when observing the progress (via the onUpload listener), it seems that all the bytes are immediately uploaded to my ktor server.

The server is localhost, but it makes an S3 upload via the S3 putObject which receives an InputStream, and this input stream is just part.streamProvider().

So I am guessing the issue is not the server (it should not read all bytes first and then upload to S3) but my client, probably append("file", bytes, does not upload in a streamed fashion (please correct me if I am wrong).

So my question is: what method can I use in a multplatform ktor client which appends a stream multipart?

[Aleksei Tirman [JB]]

Can you tell me the size of the file part?

[Rok Oblak]

I am trying with a 10MB image, so about 10_000_000 bytes; if I add this progress listener it immediately shows all 10MB as uploaded.
but my server takes another 5+ seconds to upload to S3.
Even though the server only takes the multipart file and takes the input stream and passes the inputstream to AWS SDK put request.

[Aleksei Tirman [JB]]

That's strange. On JVM, the onUpload handler in the following code is called for every 4088 bytes written:

val client = HttpClient(OkHttp)
val request = HttpRequestBuilder().apply {
    url("<https://httpbin.org/post>")
    method = HttpMethod.Post
    setBody(
        MultiPartFormDataContent(
            formData {
                append("file", File("path/to/file").readBytes())
            }
        )
    )

    onUpload { written, total ->
        println(written)
    }
}
val response = client.request(request)

[Rok Oblak]

yes, I think I have seen similar. So it was still sent to the server not in one go, but in chunks.

But then I am not sure why the server reads the entire stream at once, even though I send it to S3 as an input stream:

inputStream = part.streamProvider()

PutObjectRequest(config.bucketName, key, inputStream, metadata)

The only suspect I see is the suspending forEachPart function, which I am not sure if it suspends until it receives all parts, even though when receiving FileItem I do not do anything with it. Will debug more.