Is there a preferred way of limiting parallelism w...

# announcements

[Scott Dillender]

Is there a preferred way of limiting parallelism with coroutines? I have a sequence (A) -> lookup function (B) -> aggregate results(C). I split A into chunks and I want step B to be run in parallel(more or less a blocking database call) at most X times. I've had success using a newFixedThreadPoolContext but that's deprecated. I've also tried a fixed sized channel, but I have to create the coroutines as async(start=LAZY) and then await them one by one if I do a fold or a map.

[Scott Dillender]

Related: https://github.com/Kotlin/kotlinx.coroutines/issues/261

[araqnid]

I would simply create a fixed-size thread pool (using Executors), and use the Executor.asCoroutineContext() to assign coroutines to it. The issue you mentioned points out this introduces mandatory thread context switches onto that pool --- I don’t have anything where that would be a big enough issue right now

[araqnid]

That looks to be essentially what the deprecated newFixedThreadPoolContext does anyway

[Scott Dillender]

Thanks. I was looking for a way to not have to create the threadpool, while still establishing a limit, which seems similar to to the goal of the issue. I'll run with what's there or something similar to your suggesting. Porting to the new API should be pretty trivial when it's released. Thanks again.

[Dominaezzz]

If you use the Sequence.chunked function, you should be able to do what you describe.

[Scott Dillender]

@dominic I was able to get a form of what I wanted by chunking twice. It runs in parallel (slightly less efficiently, but I don't need to make a thread pool). Thanks https://gist.github.com/impatient/45ba44e5936f372e909f4cf4a7858b42

[dave08]

See fan in fan out pattern in the coroutines guide using Channels.

[Scott Dillender]

Thanks @dave08. Good clear example. I must have looked past that 10 times. That would have been great for a prior refactoring. I kind of want to tweak everything to make it work, but time.