Using coroutine syntax to make orchestration of AP...

# coroutines

[rrva]

Using coroutine syntax to make orchestration of API calls easier to read/write is something I found useful. I could easily see it be so useful that it should go into kotlinx-coroutines...

Here I have blocking calls, but I still find coroutine syntax elegant and simple to reason about.

Like this:

private fun <A, B> Iterable<A>.parallelMap(context: CoroutineContext, f: suspend (A) -> B): List<B> = runBlocking {
        map { async(context) { f(it) } }.map { it.await() }
    }

val listOfIds.parallelMap(fooApiPool) { fooApiClient.fetchStuff(it)}

(found at: https://jivimberg.io/blog/2018/05/04/parallel-map-in-kotlin/)

But what about orchestrating calls to several sources (clients calls could be made suspendable if this is important), where we for example have three source API:s which we would like to be called in parallel, and then the values are awaited and collected? (I'm trying to get rid of RxJava for some use-cases where I found it to be a burden to maintain/debug). What are good patterns here for expressing the execution flow?

[gildor]

Blocking parallelMap based on coroutines looks really strange. I wouldn’t use such implementation

[rrva]

it's useful for stuff which does not run in a coroutine, just using coroutines to execute some blocking stuff in parallel

[gildor]

It is not. It’s just bad style that encurage implicit conversion of non-blocking operations to blocking

[rrva]

fooApiClient.fetchStuff() in this case is a blocking operation already

[gildor]

This implementation is really bad, for example, you cannot cancel this operation, it’s really easy to shoot in your own foot if someone pass wrong context that has dispatcher that shouldn’t be blocked

[gildor]

if fooApiClient.fetchStuff is blocking, you shouldn’t create special functions to run it, but instead, at least wrap it to coroutine, so it will be non-blocking at least

[rrva]

The way I use it is with a dedicated dispatcher

[gildor]

If it work for you, fine
But you said

I meant parallelMap should go into the standard coroutine library

[gildor]

and I pointed,. that this is bad implementation that shouldn’t be used if you don’t understand all the pitfalls of it. And it’s of course shouldn’t be in kotlinx.coroutines

[gildor]

There is feature request about standard working pool pattern in kotlinx.coroutines, which related, but proposed solution doesn’t involve (of course) runBlocking or any blocking solutions

[rrva]

ok, I can see the problems with making such a function generally available.. I just thought it was an elegant way to write something a bit similar to parallelStream in syntax

[gildor]

https://github.com/Kotlin/kotlinx.coroutines/issues/172

[gildor]

Why not just use parallelStream? also parallelStream uses under the hood ForkJoinPool which has much more complicated logic and optimised for parallel blocking operations

[gildor]

so I would say not “elegant way to write parallelStream” but “naive way to write parallelStream”

[gildor]

Also, you say that fooApiClient.fetchStuff is blocking, but why it’s blocking? This example looks as some network request, which of course shouldn’t be blocking and much more optimal way would be to use non-blocking version of network request

[rrva]

Yes, agreed this could be optimized for making coroutines a more natural fit. I could have used parallelStream for this example I guess. Onto my next question, what about good patterns to orchestrate calls to multiple sources (let's say these are all HTTP calls, which I could rewrite to be non-blocking). Any examples of this?

Lets say we have the dependency that we want to fetch data in parallell from clientA, clientB, clientC, and when these three are done, we invoke clientD. In this case I would need all clients to be non-blocking and use awaitAll ?

[gildor]

I would need all clients to be non-blocking and use awaitAll ?

It’s not something what you must to do, but it’s definitely would be good in terms of performance and coroutines ideology

[gildor]

Your data sources also can be blocking (or some of them), just wrap those calls to suspend fun call(): withContext(IO) { blockingCall() }, so it allows you to use them with non-blocking code, but this approacch is also not perfect, you don’t have proper cancellation support, but it’s already closer to usual coroutine usage

[gildor]

for http requests I don’t see reasons to use blocking API with coroutines