Hi , I got tons of “ktor-client-apache” threads wh...

# ktor

[smallufo]

Hi , I got tons of “ktor-client-apache” threads when running a server. And in fact the server is not doing anything . Is there anyway to find out where goes wrong ?

jstack 45158 | grep Ktor-client-apache | wc -l                                                                                                                         
     125      // there are 125 threads generated !

And the thread number starts from #19 to #144

jstack 45158 | grep Ktor-client-apache
                                                                                                                                 
"Ktor-client-apache" #19 daemon prio=5 os_prio=31 cpu=12.09ms elapsed=513.05s tid=0x0000000135bc4c00 nid=0x8403 runnable  [0x0000000172df2000]
"Ktor-client-apache" #20 daemon prio=5 os_prio=31 cpu=11.36ms elapsed=513.05s tid=0x0000000144e30200 nid=0x7b03 runnable  [0x0000000172ffe000]
...
...
"Ktor-client-apache" #143 daemon prio=5 os_prio=31 cpu=8.96ms elapsed=510.76s tid=0x0000000142a3d400 nid=0x1fe03 runnable  [0x0000000292bea000]
"Ktor-client-apache" #144 daemon prio=5 os_prio=31 cpu=7.24ms elapsed=510.76s tid=0x0000000142a3ce00 nid=0x21903 runnable  [0x0000000292df6000]

And I cannot figure out how it starts … :

"Ktor-client-apache" #144 daemon prio=5 os_prio=31 cpu=8.90ms elapsed=591.80s tid=0x0000000142a3ce00 nid=0x21903 runnable  [0x0000000292df6000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.KQueue.poll(java.base@15.0.2/Native Method)
	at sun.nio.ch.KQueueSelectorImpl.doSelect(java.base@15.0.2/KQueueSelectorImpl.java:122)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@15.0.2/SelectorImpl.java:129)
	- locked <0x00000006c6b79ec0> (a sun.nio.ch.Util$2)
	- locked <0x00000006c6b79e68> (a sun.nio.ch.KQueueSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@15.0.2/SelectorImpl.java:141)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.lang.Thread.run(java.base@15.0.2/Thread.java:832)

Is it normal ?

I have many beans defining ktor like this :

@Named 
class SomeBean {
  private val ktorClient = HttpClient {
    followRedirects = false
    expectSuccess = false
  }
}

It seems there is engine block can override threadsCount , but I never invoked it.

Environments :

<ktor.version>1.6.1</ktor.version>

      <dependency>
        <groupId>io.ktor</groupId>
        <artifactId>ktor-client-apache</artifactId>
        <version>${ktor.version}</version>
      </dependency>

openjdk version "15.0.2" 2021-01-19
OpenJDK Runtime Environment Zulu15.29+15-CA (build 15.0.2+7)
OpenJDK 64-Bit Server VM Zulu15.29+15-CA (build 15.0.2+7, mixed mode)

These threads are all green (RUNNING) when viewing in VisualVM , but they are not transmitting anything.

[Joost Klitsie]

I guess the answer is in this message: I have many beans defining ktor like this :

[Joost Klitsie]

every client will start up threads

[Joost Klitsie]

so if you want to use less resources you have to share the client

[smallufo]

That’s also the reason I can think of. But is it true ? Every bean with Ktor client will default occupy 4 threads? Is it reasonable ? Why not initialize threads when needed or using some pool mechanism ?

[Aleksei Tirman [JB]]

You can customize a number of threads and other options using customizeClient by mutating HttpAsyncClientBuilder:

val client = HttpClient(Apache) {
    engine {
        customizeClient {
            setDefaultIOReactorConfig(
                IOReactorConfig.custom()
                    .setIoThreadCount(Runtime.getRuntime().availableProcessors())
                    .build()
            )
        }
    }
}

[smallufo]

Thanks @Aleksei Tirman [JB] . I tried another way , Initialize ktor centrally (in one spring @Configuration class) , and inject ktor’s io.ktor.client to each component. And it reduces the total threads from 125 to 15 . It seems each ktor initialization will consume 5 threads. Anyway , is there any drawback about this ? Thanks.

[Aleksei Tirman [JB]]

There should be none.

[smallufo]

If I reduce total client instance to 1 (means 5 threads , maybe) , does it mean my server (including many api beans & queries) will only be sending 5 http request at the same time ? Thanks.

[Aleksei Tirman [JB]]

I don't think so. The following simple test shows that all requests are executed concurrently.

(1..50).map {
    async {
        client.get<String>("<https://httpbin.org/get>")
        println("$it done")
    }
}.awaitAll()

[smallufo]

BTW , It seems that intelliJ cannot recognize HttpClient(Apache) . It reports error , so that I cannot use apache’s specific settings ( ex : customizeClient ) . What’s the cause ?

Thanks.

[Aleksei Tirman [JB]]

You should use engine property to configure the engine's specific options. Please see https://kotlinlang.slack.com/archives/C0A974TJ9/p1627466069105900?thread_ts=1627379761.098200&cid=C0A974TJ9.

[smallufo]

Yes , I am copying the same code . But intelliJ just cannot recognize HttpClient(Apache) as below :
It seems kotlin’s compiler is broken here.

[Aleksei Tirman [JB]]

That's probably an IDE problem.