Hello everyone. I think I have some memory leak in...

# server

[Rafał Kuźmiński]

Hello everyone.
I think I have some memory leak in my Ktor based application. Tomcat is spawning many Threads and never releases them. Is there any way to hook into moment when Ktor requests new Thread? Some parts of application are working fine (I can create hundreds requests and none Tomcat thread is created). Or maybe this is bad idea and I should log all currently working threads? Or do you suggest any other way of checking this issue?

[Rafał Kuźmiński]

When I checked logs I can see that bouth OkHttp client and DefaultDispatcher threads aren't closed:
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [treespot-dashboard##0.4.4] appears to have started a thread named [DefaultDispatcher-worker-10] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:357)
kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.park(CoroutineScheduler.kt:783)
kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.tryPark(CoroutineScheduler.kt:728)
kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:711)
kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
sorg.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [treespot-dashboard##0.4.4] appears to have started a thread named [OkHttp Dispatcher] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
java.base@11.0.9.1/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
java.base@11.0.9.1/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
java.base@11.0.9.1/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
java.base@11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@11.0.9.1/java.lang.Thread.run(Thread.java:834)

[tddmonkey]

Why do you think you have a memory leak? Purely based on what you have observed from your threads?

[Rafał Kuźmiński]

Yes, I have no other indicator (but i am not that much experience in server development). If I would refresh my web app like 1000 times, Tomcat would stop responding, because it wouldn't be able to create new Thread (there is a limit for ~4000 threads)

[tddmonkey]

What do you mean by refresh? Do you mean hot-reload your webapps?

[Rafał Kuźmiński]

Ah no, sorry I didn't describe it well enough. This web application have some frontend, that is connecting with backend through HTTP. After i hit refresh on browser, there a several requests to backend. That's why after 1000 refreshes, server would be full.

[Joost Klitsie]

So which part of the application is spwaning the threads? Server or Client?

[Joost Klitsie]

if it is the client (which I will assume as you mention OKHttp) are you reusing the client or all the time creating a new one?

[Rafał Kuźmiński]

@Joost Klitsie I am not really sure. That would be first thing that I have to check, but I am not sure how. I just checked Dependency container settings and HttpClient is created every time insted just once, so maybe this is an issue.

[Joost Klitsie]

I would say so

[Rafał Kuźmiński]

Yes, it was it. Thank you very much, I had different HttpClient for OAuth and it was created every time for authorization. Thanks 😉

[Rafał Kuźmiński]

@Joost Klitsie could you maybe tell me why this was happening? Or point me to some docs that would explain why more than one HttpClient is a problem?

[Joost Klitsie]

I am unsure as to when httpclients release their resources, so spawning multiple will cost you in terms of initialization time and also you will probably have to do the clean up yourself. Internally a client has a connection pool and will reuse this across different requests, making it more efficient firing multiple calls to the same host. So I would say the optimized way of reusing connections and of course the threads a client will spawn are a good reason to reuse a client throughout your application

[Rafał Kuźmiński]

I see. I just found official docs about it (https://ktor.io/docs/client.html) and you are right, releasing resources should be done explict in code