Hi one of our servers crashed last night with this error ```

Question

Hi one of our servers crashed last night with this error ```Failed to start thread pthread create failed EAGAIN for attributes stacksize 512k guardsize 0k detached Exception in thread HTTP listener 11490 1 java lang OutOfMemoryError unable to create native thread possibly out of memory or process resource limits reached at java base java lang Thread start0 Native Method at java base java lang Thread start Thread java 803 at java base java util concurrent ThreadPoolExecutor addWorker ThreadPoolExecutor java 937 at java base java util concurrent ThreadPoolExecutor processWorkerExit ThreadPoolExecutor java 1005 at java base java util concurrent ThreadPoolExecutor runWorker ThreadPoolExecutor java 1142 at java base java util concurrent ThreadPoolExecutor$Worker run ThreadPoolExecutor java 628 at java base java lang Thread run Thread java 834 ``` It definitely wasn t running out of memory We do spawn a lot of coroutines however my understanding is that shouldn t be an issue The one place we use coroutines is ``` val pushNotificationService = this pushNotificationService GlobalScope launch pushNotificationService sendPushKit id ``` Any ideas what could be going wrong Thanks

bdawg.io · Answer

I m not seeing anything in your shared exception and stack trace that s suggesting it s coroutines

krtko · Answer

Ok thanks

hho · Answer

Exactly Your server ran out of threads while trying to extend a ThreadPoolExecutor You should probably check if you have any unbounded thread pools in your app or if you re on Linux if your machine is configured to allow enough processes

krtko · Answer

I m on Linux Its deployed via Heroku s default JVM runtime container

gildor · Answer

not sure why do you think that it is not OOM it s definitely is and I agree most probably because of big amount of threads

krtko · Answer

Our memory monitor shows there still being at about 60 of total memory and only about 40 of the heaps total size

gildor · Answer

gt possibly out of memory or process resource limits reached

gildor · Answer

you could create just too many threads