What’s the easiest way of achieving parallelization for something like

myList.map { myCpuIntensiveTask(it) }

?

myList.parallelStream().map { myCpuIntensiveTask(it) }.collect(Collectors.toList())

But we plan to provide simple, declaration and expressive API to do some like that with coroutines in the future, too.

Cool. Are the plans in some issue tracker? I’d love to follow the discussion

@elizarov will that be the future design of kx.coroutines, more declarative APIs like the ones in other libs? Fewer imperative-like code

so basing it off iterables with parallel properties, like in Java 8. Neat!

No. Completely unlike Java 8. Much neater.

What is the timeline for this?

Flow ~= asynchronous Sequence, so that's why we use sequences API as an inspiration a lot

hot (hate that terminology, it’s eager and lazy) and single use is what deferred and job are by default anyway

multi-shot continuations will need separation from immutable and mutable state on a language level which is an extremely hard thing to design properly in a language that supports mutable state (as in "there is no existing science that gives clue to a pragmatic answer"), so no, they are not coming in a near future.

Is this a decent approach?

myList.map {
    async { myCpuIntensiveTask(it) }
}.map {
    it.await()
}

.map { it.await() }

==

.awaitAll()

. Otherwise, it is Ok, but there is no limit on concurrency in this solution, with potentially disastrous consequences at scale.

What are the potentially disastrous consequences?

In case of "cpu intensive task" -- Running out of memory

This should be parallel given that I’m using Dispatchers.Default. It is creating more coroutines than CPU cores, but they are still multiplexed onto num CPU cores number of threads, right?

Yes.

Ah, but you mean I risk run out of memory because I’m creating lots of coroutines, or do you mean there is risk also without coroutines?

Yes. You are creating tons of coroutines upfront/

So a better implementation would distribute the task across fewer coroutines? And the specific memory management might depend on the type of task (eg. a few heavy ones or billions of light ones). Is that the core point?

Yes. A better implementation would limit concurrency (a number of created coroutines), so for CPU intensive task you could set it to the number CPU cores, but for other tasks (like network intensive) you can use other consideration (such as scalability of downstream services you are touching)

I see now 🙂 Thank you @elizarov for giving such clear answers. Really appreciated! ⭐