https://kotlinlang.org logo
Title
k

krtko

05/15/2019, 4:50 PM
Hi one of our servers crashed last night with this error:
Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 512k, guardsize: 0k, detached. 
Exception in thread "HTTP-listener-11490-1" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached 
at java.base/java.lang.Thread.start0(Native Method)

at java.base/java.lang.Thread.start(Thread.java:803) 
at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937) 
at java.base/java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1005) 
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
at java.base/java.lang.Thread.run(Thread.java:834)
It definitely wasn't running out of memory. We do spawn a lot of coroutines, however my understanding is that shouldn't be an issue. The one place we use coroutines is:
val pushNotificationService = this.pushNotificationService
GlobalScope.launch {
    pushNotificationService.sendPushKit(id)
}
Any ideas what could be going wrong? Thanks
b

bdawg.io

05/15/2019, 4:52 PM
I'm not seeing anything in your shared exception and stack trace that's suggesting it's coroutines
k

krtko

05/15/2019, 4:54 PM
Ok thanks
h

hho

05/15/2019, 5:02 PM
Exactly. Your server ran out of threads while trying to extend a ThreadPoolExecutor. You should probably check if you have any unbounded thread pools in your app or (if you're on Linux) if your machine is configured to allow enough processes.
k

krtko

05/15/2019, 5:32 PM
I'm on Linux. Its deployed via Heroku's default JVM runtime container.
g

gildor

05/16/2019, 12:06 AM
not sure why do you think that it is not OOM, it's definitely is, and I agree most probably because of big amount of threads
k

krtko

05/16/2019, 12:38 AM
Our memory monitor shows there still being at about 60% of total memory and only about 40% of the heaps total size
g

gildor

05/16/2019, 1:00 AM
possibly out of memory or process/resource limits reached
you could create just too many threads