Since upgrading ktor from 3.4.0 to 3.4.1 or 3.4.2,...

# ktor

[Dominik Sandjaja]

Since upgrading ktor from 3.4.0 to 3.4.1 or 3.4.2, we have the issue that after a short while, within 5-10 minutes, the server stops responding to requests. First partially, then fully.
The ingress nginx logs errors like

2026/04/01 13:12:41 [error] 29#29: *1762684 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.156.0.63, server: ...

We did see this before, but only rarely, with 3.4.1 and .2, this happens reliably after a few minues.
It is not a high-load system, we see at most a few requests per second.

Rolling back to 3.4.0 fixes the issue, those pods do not run into any unresponsiveness.

Any hints what I could do to investigate this further?

[Dominik Sandjaja]

Server is running on a Java 21 runtime image, so it's fully JVM based.

[Dominik Sandjaja]

There is nothing in the server logs.

[Bruce Hamilton]

Perhaps you could try killing it with a thread dump. Once when it's responsive and once when it's not responsive. This ought to show where threads are being blocked.

[Dominik Sandjaja]

I did that a while back but the dump did not reveal anything in particular. I will try again with 3.4.2, but because the unresponsiveness happens so quickly it has actual consequences for our system. Will try, though, thanks for the hint!

[Bruce Hamilton]

If you can get a small reproducer with the problematic endpoint and some repeated calls that would be handy for investigation. You might also try changing the server engine to see if it's isolated to Netty or whichever you're on.

[Dominik Sandjaja]

If I could isolate it to a specific set of endpoints, I'd be happy to do so, but all endpoints start responding reliably after a few minutes, even health and metrics. And yes, I could try switching to Jetty (from Netty), just to check 🙂
If I could isolate it to a specific set of endpoints, I'd be happy to do so, but all endpoints stop responding reliably after a while, even

[Aleksei Tirman [JB]]

The problem might be caused by the fix for KTOR-9334. You may want to use the debug agent to see all active coroutines and their stack traces.

[Dominik Sandjaja]

@Aleksei Tirman [JB]: Thank you! I have the hypothesis that it is related to some blocking actions on coroutines where it shouldn't happen and I already created dedicated <http://Dispatchers.IO|Dispatchers.IO>.limitedParallelism() dispatchers to remove as many potential troublemakers as I could, but the debug agent sounds like a good thing to try next!