We've started seeing a condition on our Ktor microservices where all attempted requests hang indefinitely without returning response. On dumping stack traces for the app, we see a thread that appears to be stuck parsing JSON:
https://gist.github.com/mattrussell-sonocent/6a6862354cfa7728b708318ba75eb536
Our corresponding app code is:
val credentials = call.receive<Credentials>()
A
netstat -a
reveals a whole bunch of other waiting HTTP connections that don't get a chance to be served.
After 5-10 minutes, the server recovers without intervention.
Any ideas on what's going on or how to diagnose?