is there somewhere an example for parallel file upload with

Question

is there somewhere an example for parallel file upload with ktor client

bitkid · Answer

i am somehow stuck a bit i have a suspend function `importFile file client submitFormWithBinaryData bla ` and a function `importDirectory dir dir listFiles forEach importFile it ` i want to upload the files in parallel do some processing with the result returned from the server best would be if i also had control over the level of concurrency nr of threads

bitkid · Answer

anyone aware of some example code going in that direction

bitkid · Answer

i came up with something like this

bitkid · Answer

```coroutineScope filesToUpload map async importFile it repositoryId map it await forEach handleResult ```

bitkid · Answer

does that make sense

Dominaezzz · Answer

The only thing I ll say is to use `awaitAll ` instead of `map it await ` You can read the doc to find out why it s better

bitkid · Answer

< Dominaezzz> do you have more experience with the ktor http client i try to upload around 30k files with around 3mb each and the client always dies on me with weird exceptions java util concurrent TimeoutException Connection lease request time out as an example

bitkid · Answer

i tried different clients jetty apache and they all die with some bullshit exceptions it s quite frustrating

bitkid · Answer

i also tried playing around with things like threadsCount setMaxConnTotal connectTimeout etc

bitkid · Answer

nothing helps

bitkid · Answer

a stupid simple python script or curl works though without problems

Dominaezzz · Answer

Hmm what happens when you upload sequentially

bitkid · Answer

without async you mean

Dominaezzz · Answer

Yes

bitkid · Answer

this i haven t tried but i mean what s the point then in using an async client

bitkid · Answer

but let me check

Dominaezzz · Answer

It might not be the client it might be the server This is just a debugging process

bitkid · Answer

server is also written with ktor

bitkid · Answer

seems to work without async

bitkid · Answer

the code which seems to work looks like this

bitkid · Answer

```filesToUpload map importFile it repositoryId forEach resp gt when resp first HttpStatusCode OK gt File statusDirectory resp third nameWithoutExtension + ok createNewFile else gt File statusDirectory resp third nameWithoutExtension + fail writeText resp second ```

bitkid · Answer

what doesnt work is

bitkid · Answer

```coroutineScope filesToUpload map async httpClient coroutineContext importFile it repositoryId map it await forEach resp gt when resp first HttpStatusCode OK gt File statusDirectory resp third nameWithoutExtension + ok createNewFile else gt File statusDirectory resp third nameWithoutExtension + fail writeText resp second ```

bitkid · Answer

oh now also the one without async failed again with a different exception

bitkid · Answer

java lang IllegalStateException Unexpected request state READY at org apache http util Asserts check Asserts java 46 at org apache http nio protocol HttpAsyncRequestExecutor outputReady HttpAsyncRequestExecutor java 231

bitkid · Answer

arggghhh

bitkid · Answer

i hate it

bitkid · Answer

i have seen this error before Unexpected request state READY even posted it here a while again no one had an idea

bitkid · Answer

i added some retry logic now it seems to be stable but still very annoying

Jorge R · Answer

Hi Sascha this seems like a downloader uploader patter as it is explained by Roman Elizarov in last KotlinConf 2018

Jorge R · Answer

did you have a look at his implementation

Jorge R · Answer

Jorge R · Answer

I believe it is focused in Frontend but you may be able to extract some hints for Ktor

Jorge R · Answer

I hope this helps

bitkid · Answer

cool thx i ll take a look

bitkid · Answer

doesnt look to me like its focussed on frontend just general coroutines stuff but yeah maybe i try to rewrite my code with channels maybe that makes it more clear