Hi one of our servers crashed last night with this...

# coroutines

[krtko]

Hi one of our servers crashed last night with this error:

Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 512k, guardsize: 0k, detached. 
Exception in thread "HTTP-listener-11490-1" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached 
at java.base/java.lang.Thread.start0(Native Method)

at java.base/java.lang.Thread.start(Thread.java:803) 
at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937) 
at java.base/java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1005) 
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
at java.base/java.lang.Thread.run(Thread.java:834)

It definitely wasn't running out of memory. We do spawn a lot of coroutines, however my understanding is that shouldn't be an issue. The one place we use coroutines is:

val pushNotificationService = this.pushNotificationService
GlobalScope.launch {
    pushNotificationService.sendPushKit(id)
}

Any ideas what could be going wrong? Thanks

[bdawg.io]

I'm not seeing anything in your shared exception and stack trace that's suggesting it's coroutines

[krtko]

Ok thanks

[hho]

Exactly. Your server ran out of threads while trying to extend a ThreadPoolExecutor. You should probably check if you have any unbounded thread pools in your app or (if you're on Linux) if your machine is configured to allow enough processes.

[krtko]

I'm on Linux. Its deployed via Heroku's default JVM runtime container.

[gildor]

not sure why do you think that it is not OOM, it's definitely is, and I agree most probably because of big amount of threads

[krtko]

Our memory monitor shows there still being at about 60% of total memory and only about 40% of the heaps total size

[gildor]

possibly out of memory or process/resource limits reached

[gildor]

you could create just too many threads