We have some HTTP requests which we are currently ...

# coroutines

[Jonathan Mew]

We have some HTTP requests which we are currently running consecutively but it's taking too long, so we would like to parallelise them. We would need to halt execution until they are all complete, and propagate an exception if any failed (wouldn't matter which exception was thrown if multiple requests fail). We don't need to capture any response data.

What would be the neatest/most idiomatic way to achieve this?

[Jonathan Mew]

(I suspect I could get something working, but reckon this domain has some useful gadgets that might be easy to miss...)

[streetsofboston]

allMyRequests.map { request -> async { request.invoke(input) } }.awaitAll()

[Jonathan Mew]

Thank you! My initial stab was to correctly map to an async {} block, but wasn't aware of awaitAll() 👍

[streetsofboston]

If you don’t have any results, use launch and joinAll

[Jonathan Mew]

would that still propagate an exception?

[streetsofboston]

Yup. The top-most launch (myScope.launch { …. }) will handle it and if you have a CoroutineExceptionHandler installed int that top-most scope, it will receive it.

[streetsofboston]

https://medium.com/the-kotlin-chronicle/coroutine-exceptions-3378f51a7d33
(shameless plug 🙂 )

[Jonathan Mew]

So we gave the launch/joinAll a try and it ran fine but didn't speed things along after all. And I realise now that obviously it wouldn't, because we are doing blocking I/O 🤦

We are using the AWS Java SDK to interact with S3, and would prefer not to reimplement it. In that case, I guess we are back to using a classic thread pool? Or does Kotlin still offer something to field that side of things for us?

[streetsofboston]

If you use Dispatchers.IO, you get to use a pool of about 64 threads (on the JVM) (it borrows from the Dipatchers.Default if it runs out of threads, iirc).
If you need to control it your self, on the JVM, you can easily get a Dispatcher from a Java Executor that is configured with your own thread-pool management.

[Jonathan Mew]

thank you!