We are looking into an issue where our CPU usage g...

# coroutines

[nomad]

We are looking into an issue where our CPU usage goes way up when switching from an old version of ktor to a new one. We have profiled extensively, and cannot find any code in ktor that leads to this performance regression. All we can think of is ktors increasing use of coroutines since version 1.0+, all of which show this regression.

I have attached a screen dump from our profiler, I am mainly wondering if it is expected that we spend ~30% in the CoroutineScheduler$Worker.run method? When profiling the older versions of ktor, we see the same performance from the other methods just without this ~30% chunk. Is this unavoidable overhead from coroutines? A bad reporting from the profiler? or am I just missing something?

We are also seeing a big increase in kernel mode cpu usage, nearly doubling with the same workload on ktor 1.0+ versions.

This is with ktor 1.2.3 on kotlin 1.3.41

[ribesg]

You should open an issue about this on the ktor repository on Github, with some more details like what versions you are comparing

[nomad]

I am posting here because the 30% is a coroutine method, and I see very, very little ktor in my profiling result. My question is whether this is expected or not.

[altavir]

What profiler do you use. I think that picture shows that most of the time you are inside a coroutine. That is all.

[nomad]

jprofiler

[altavir]

Also this picture shows that most of the time you are spending on building a tree using kotlinx.html

[nomad]

No, it shows that i am spending ~30% in the method I am inquiring about.

[louiscad]

You are probably encountering this issue that should be fixed in next version (1.3.2): https://github.com/Kotlin/kotlinx.coroutines/issues/1046#issuecomment-531165477

[altavir]

I mean that self time could be a bit confusing and different profilers interpret it differenrly. If you are have method that just launches blocking operation and waits for it to finish, than you will spend a lot of time in it. If you just call some method inside run, profiler could catch it. But if you have, say inline function there, it is not clear. I do not say, that there is no problem. Just that is not always this strightforward.

[groostav]

I'm also using EJ JProfiler on a desktop app and I'm pretty sure JProfiler does not report "in a coroutine" as per @altavir's suggestion. I'm reasonably certain of this because the difference in profiler image is night and day when using launch and run-blocking/manual-event-loop. As far as the compiled code is concerned, a suspend fun x() becomes fun x$1() and fun x$2(), which the profiler sees, dutifully profiles, watch the resulting someAPI.addListener go off at the suspension point, and then spam-return to the parent event loop. fun x$1 and fun x$2, the before & after suspension-point parts of your suspend fun x dont have anything to do with each other as far as JProfiler is concerned.

[groostav]

maybe this is presumptuous, but can I poke @ikej here?