is there somewhere an example for parallel file upload with ktor client?

# ktor

[bitkid]

is there somewhere an example for parallel file upload with ktor client?

[bitkid]

i am somehow stuck a bit .. i have a suspend function importFile(file) {client.submitFormWithBinaryData(bla)} and a function importDirectory(dir) {dir.listFiles().forEach{importFile(it)}} .. i want to upload the files in parallel, do some processing with the result returned from the server. best would be if i also had control over the level of concurrency (nr. of threads?)

[bitkid]

anyone aware of some example code going in that direction?

[bitkid]

i came up with something like this

[bitkid]

coroutineScope {
            filesToUpload.map { async { importFile(it, repositoryId) } }.map { it.await() }.forEach { handleResult() }
        }

[bitkid]

does that make sense?

[Dominaezzz]

The only thing I'll say is to use awaitAll() instead of map { it.await() }.
You can read the doc to find out why it's better.

[bitkid]

@Dominaezzz do you have more experience with the ktor http client? i try to upload around 30k files with around 3mb each .. and the client always dies on me with weird exceptions. java.util.concurrent.TimeoutException: Connection lease request time out as an example

[bitkid]

i tried different clients (jetty, apache) and they all die with some bullshit exceptions. it's quite frustrating

[bitkid]

i also tried playing around with things like threadsCount, setMaxConnTotal, connectTimeout etc

[bitkid]

nothing helps

[bitkid]

a stupid simple python script or curl works though without problems

[Dominaezzz]

Hmm, what happens when you upload sequentially?

[bitkid]

without async{} you mean?

[Dominaezzz]

Yes

[bitkid]

this i haven't tried .. but i mean what's the point then in using an async client?

[bitkid]

but let me check

[Dominaezzz]

It might not be the client, it might be the server. This is just a debugging process.

[bitkid]

server is also written with ktor

[bitkid]

seems to work without async

[bitkid]

the code which seems to work looks like this

[bitkid]

filesToUpload.map { importFile(it, repositoryId) }.forEach { resp ->
            when (resp.first) {
                HttpStatusCode.OK -> File(statusDirectory, resp.third.nameWithoutExtension + ".ok").createNewFile()
                else -> File(statusDirectory, resp.third.nameWithoutExtension + ".fail").writeText(resp.second)
            }
        }

[bitkid]

what doesnt work is

[bitkid]

coroutineScope {
            filesToUpload.map { async(httpClient.coroutineContext) { importFile(it, repositoryId) } }.map { it.await() }.forEach { resp ->
                when (resp.first) {
                    HttpStatusCode.OK -> File(statusDirectory, resp.third.nameWithoutExtension + ".ok").createNewFile()
                    else -> File(statusDirectory, resp.third.nameWithoutExtension + ".fail").writeText(resp.second)
                }
            }
        }

[bitkid]

oh .. now also the one without async failed .. again with a different exception

[bitkid]

java.lang.IllegalStateException: Unexpected request state READY
at org.apache.http.util.Asserts.check(Asserts.java:46)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.outputReady(HttpAsyncRequestExecutor.java:231)

[bitkid]

arggghhh

[bitkid]

i hate it

[bitkid]

i have seen this error before ..Unexpected request state READY.. even posted it here a while again .. no one had an idea

[bitkid]

i added some retry logic now it seems to be stable .. but still very annoying

[Jorge R]

Hi Sascha, this seems like a downloader/uploader patter, as it is explained by Roman Elizarov in last KotlinConf 2018

[Jorge R]

did you have a look at his implementation?

[Jorge R]

https://www.youtube.com/watch?v=a3agLJQ6vt8▾

[Jorge R]

I believe it is focused in Frontend, but you may be able to extract some hints for Ktor?

[Jorge R]

I hope this helps

[bitkid]

cool thx .. i'll take a look

[bitkid]

doesnt look to me like its focussed on frontend .. just general coroutines stuff .. but yeah. maybe i try to rewrite my code with channels .. maybe that makes it more clear.