Is all that parking overhead from coroutines? Are there approaches to get that down? Looks like that 50% of the application’s CPU time is spent on locking.

IO-bound coroutines should be run on the Dispatchers.IO dispatcher which is specifically targeted to be used in blocking (io) code, where threads will be doing a lot of waiting/parking.

Yeah that’s what I’m doing. But there may be unintentional switches between Default and IO or Main and IO etc. Finding those would be helpful, should there be some. Mistakes happen 🙂

What do you mean by unintentional switches? Is it important on which thread (if it is jvm/native) your code is run?

Dispatcher switches, not Thread switches. Dispatcher switches aren’t that cheap for coroutines afaik, at least not at high volume.

Ah, I see, performance wise. I know a coroutine’s resumption, after a suspension, has an affinity to the same dispatcher and even thread. But I can’t comment on the exact performance impact on this…. I don’t know enough about it. 🙂

Yeah me too. I’m trying to find info on that.

Last time I looked through the scheduler: CPU -> IO is free because you stay on the same thread, it just gives up it’s CPU lease. IO -> CPU may incur a park and thread switch if it can’t acquire a CPU lease in time

I see. So basically I need to check for too many

flowOn

,

launch(In)

and

withContext()

calls that delegate to Default dispatcher where it’s not strictly necessary. I’ll check the coroutines code. Maybe I Can put a breakpoint somewhere and get an idea how often it happens and where.

You know, I see the same unpark overhead here - CPU or bound worker, it doesn’t matter. I’ve been tinkering with a reactor flux as a flow, quite a bit of CPU overhead measured both with sampling and async profiler

What do you use for profiling?

unintentional switches between Default and IO

Default and IO share threads, so there is no switch between them

Marc - that’s YourKit

so there is no switch between them

I think you can rely on it only in one direction, when switching from Default to IO. When you're switching from IO to Default back, I think it's very likely that there will be thread switch.

why do you think so?

let's say you have 64 running coroutines in IO, what will happen if all of them will switch to Default dispatcher and call blocking call?

@bezrukov Your code has race conditions. You need to specify which coroutines is printing its thread.

is it? the only async piece of code is

repeat(8) {

launch(Dispatchers.Default) {

Thread.sleep(50) // blocking call

}

}

that doesn't print anything. All other print statements are "sequential" in terms of coroutines

Ah yes, you're right, looked too fast.

it's absolutely the same as Andrey's example, just with quoted code mentioned above. It just makes Default dispatcher "busy"

Correct me if I am wrong: coroutines on jvm are just built on top of executors and Runnable. It takes a suspend method and splits into suspension points with a Runnable for each split and puts it in queues which are used like an event loop and may context switched when pulled off queue. I appreciate someone quantifying the overhead.

I’m actually not sure whether my app was IO or CPU bound. Both usages were pretty high. I’ve done a lot of optimizations by now and am currently recording with Java Flight Recorder to check if that yields different results. Then again with CPU profiling. If it’s IO bound I would expect a lot of parking. But not parking by coroutines but by the actual IO operation, e.g. in the HTTP client or the database client.

i have an app that does millions of http requests (using netty), a little xml parsing, I look at total server cpu usage it is 5-10%. I do application profiling most time (over 50%) is spent on coroutines like you show. My conclusion everything is as expected.

Yeah it may also be nothing at all, just the run loop waiting for work.

Maybe parking is done for longer periods of time, eating less CPU?

Yeah, maybe I had hit a big inefficiency before, where the dispatchers were almost always busy and now the thread are rarely exhausted so they can rest in peace ¯\_(ツ)_/¯

Is your application open source? I would be interested in looking at code. If those were threads I would suspect over use of concurrency / creating too many threads. There is such a thing as using too many threads where the overhead of switching threads out weights the benefits of having so many and you should reduce the limit. With coroutines that is way more unlikely though.

I'm afraid it's not 😅 I also wonder of its the amount of Flows I've created (50k+). But most of them will just idle in

delay

i've had problems with using delay before. I have a project where I was using stomp protocol, which is persistent and was sending a ping every 15 seconds on each connection. I had millions of connections using delay in a while(isActive) to send pings. Turns out that didn't work, after a certain number of connections there was too much overhead in the creating of so many coroutines where the delays weren't accurate anymore. I had to change the design to scale better by keeping track of when the last ping was sent based on timestamp for each connection basically not using delay.

Are you sure the delays weren’t accurate? Maybe too many coroutines were waking up at the same time which caused all coroutine resumption to have significant backpressure.

I knew the delays weren't accurate because that was the client side, on the server side I had multiple servers (6 so less pressure on server side because it is spread) and they detect if a ping didn't arrive on time and close connection after 30 seconds, double the delay time, I would then ramp up the connections and start seeing disconnect after 400k connections. After profiling and doing a memory dump I noticed significant coroutine overhead, and I just changed design to not use delay and then I didn't have issues.

Maybe worth to report it and see what kotlinx.coroutinws team will answer, there might be some bottle neck in dispatcher implementation

Could also be sampling profilers misreporting incredibly numerous but very short calls - using a tracing profiler instead will give more accurate results