https://kotlinlang.org logo
#ktor
Title
# ktor
b

bitkid

10/01/2019, 11:15 AM
is there somewhere an example for parallel file upload with ktor client?
i am somehow stuck a bit .. i have a suspend function
importFile(file) {client.submitFormWithBinaryData(bla)}
and a function
importDirectory(dir) {dir.listFiles().forEach{importFile(it)}}
.. i want to upload the files in parallel, do some processing with the result returned from the server. best would be if i also had control over the level of concurrency (nr. of threads?)
anyone aware of some example code going in that direction?
i came up with something like this
Copy code
coroutineScope {
            filesToUpload.map { async { importFile(it, repositoryId) } }.map { it.await() }.forEach { handleResult() }
        }
does that make sense?
d

Dominaezzz

10/01/2019, 12:15 PM
The only thing I'll say is to use
awaitAll()
instead of
map { it.await() }
. You can read the doc to find out why it's better.
b

bitkid

10/02/2019, 12:45 PM
@Dominaezzz do you have more experience with the ktor http client? i try to upload around 30k files with around 3mb each .. and the client always dies on me with weird exceptions. java.util.concurrent.TimeoutException: Connection lease request time out as an example
i tried different clients (jetty, apache) and they all die with some bullshit exceptions. it's quite frustrating
i also tried playing around with things like threadsCount, setMaxConnTotal, connectTimeout etc
nothing helps
a stupid simple python script or curl works though without problems
d

Dominaezzz

10/02/2019, 12:48 PM
Hmm, what happens when you upload sequentially?
b

bitkid

10/02/2019, 12:48 PM
without async{} you mean?
d

Dominaezzz

10/02/2019, 12:48 PM
Yes
b

bitkid

10/02/2019, 12:49 PM
this i haven't tried .. but i mean what's the point then in using an async client?
but let me check
d

Dominaezzz

10/02/2019, 12:50 PM
It might not be the client, it might be the server. This is just a debugging process.
b

bitkid

10/02/2019, 12:51 PM
server is also written with ktor
seems to work without async
the code which seems to work looks like this
Copy code
filesToUpload.map { importFile(it, repositoryId) }.forEach { resp ->
            when (resp.first) {
                HttpStatusCode.OK -> File(statusDirectory, resp.third.nameWithoutExtension + ".ok").createNewFile()
                else -> File(statusDirectory, resp.third.nameWithoutExtension + ".fail").writeText(resp.second)
            }
        }
what doesnt work is
Copy code
coroutineScope {
            filesToUpload.map { async(httpClient.coroutineContext) { importFile(it, repositoryId) } }.map { it.await() }.forEach { resp ->
                when (resp.first) {
                    HttpStatusCode.OK -> File(statusDirectory, resp.third.nameWithoutExtension + ".ok").createNewFile()
                    else -> File(statusDirectory, resp.third.nameWithoutExtension + ".fail").writeText(resp.second)
                }
            }
        }
oh .. now also the one without async failed .. again with a different exception
java.lang.IllegalStateException: Unexpected request state READY at org.apache.http.util.Asserts.check(Asserts.java:46) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.outputReady(HttpAsyncRequestExecutor.java:231)
arggghhh
i hate it
i have seen this error before ..Unexpected request state READY.. even posted it here a while again .. no one had an idea
i added some retry logic now it seems to be stable .. but still very annoying
j

Jorge R

10/03/2019, 11:10 AM
Hi Sascha, this seems like a downloader/uploader patter, as it is explained by Roman Elizarov in last KotlinConf 2018
did you have a look at his implementation?

https://www.youtube.com/watch?v=a3agLJQ6vt8

I believe it is focused in Frontend, but you may be able to extract some hints for Ktor?
I hope this helps
b

bitkid

10/03/2019, 11:20 AM
cool thx .. i'll take a look
doesnt look to me like its focussed on frontend .. just general coroutines stuff .. but yeah. maybe i try to rewrite my code with channels .. maybe that makes it more clear.
👍 1
9 Views