For some reason sometimes my ktor http client (usi...

# ktor

[MrPowerGamerBR]

For some reason sometimes my ktor http client (using Apache) gets stuck (no matter what request I do, it just timeouts) after a while the pending requests are executed.

What I'm thinking it is happening is that it is queueing requests instead of executing them... how can I change how many parallel requests can I execute with the same client instance? (unless if I should create a new http client every time... but I don't think this is needed, right?)

[MrPowerGamerBR]

And yes, I already tried doing this:

// Maximum number of socket connections.
				this.setMaxConnTotal(100_000)

				// Maximum number of requests for a specific endpoint route.
				this.setMaxConnPerRoute(100_000)

...but it didn't fix the issue 😞

[MrPowerGamerBR]

There is a

http.engine.config.threadsCount

which by default is set to 4, but I'm not sure that this is the variable that should be changed.

[e5l]

Hi @MrPowerGamerBR, what is your version of ktor?

[bitkid]

sounds similar to what i am experiencing .. with 1.3.0

[MrPowerGamerBR]

@e5l ktor 1.2.6, I can update if needed (I didn't notice that 1.3.0 was released, whoops)

To clarify, the requests are executed... eventually... and even with a timeout set it just takes waaaaaay too long to fail (that's why I think it is queueing requests, the Apache client timeout is only applied after it starts executing the request)

This only happens sometimes and I noticed that this happens only when the application is sending a lot of http requests. After a while the bug goes away and everything is back to normal.

[MrPowerGamerBR]

It isn't the coroutine dispatcher itself (created in my application with a cached thread pool) that is running out of threads because other stuff that relies on coroutines work fine

[e5l]

We fixed similar issue in 1.3.0, could you try to update?

[MrPowerGamerBR]

@e5l sure! I will try updating and if it is fixed (or not 😞 ) I will tell you. 🙂

[MrPowerGamerBR]

May take a while because well... this issue is very rare (only happens in one of my instances and happens sometimes after restarts during peak hours, rarely happens after the bootup)

I think this happens because the instance that has the issue is the main instance and it receives http callbacks from other services (like YouTube videos notifications) and relays the request to my other five instances. Because Google/YT keeps sending notifications if the instance is down, after the instance restarts it receives a lot of notifications and relays them to the other instances... causing that issue I described above where

[MrPowerGamerBR]

@e5l updating to ktor 1.3.0 didn't fix the issue 😞

[MrPowerGamerBR]

I decided to see if increasing the threadsCount would fix the issue

val http = HttpClient(Apache) {
        this.expectSuccess = false

        engine {
            this.socketTimeout = 25_000
            this.connectTimeout = 25_000
            this.connectionRequestTimeout = 25_000

            customizeClient {
                // Maximum number of socket connections.
                this.setMaxConnTotal(100_000)

                // Maximum number of requests for a specific endpoint route.
                this.setMaxConnPerRoute(100_000)
            }
        }
    }.apply {
        this.engineConfig.threadsCount = 128
    }

...and it looks like it fixed the issue, or well, I wasn't able to reproduce it now.

Nevermind, the issue still persists. I'm trying to get a thread dump with jstack when the issue happens to see if the threads are all stuck when this happens (or not).

[MrPowerGamerBR]

I will try investing the issue a bit more, it seems very strange and to be honest... I just can't find any clues pointing to why it breaks sometimes. (I thought that maybe the webserver is getting stuck but not the client... but accessing the same endpoint with the browser or any other JVM instance works fine)

I could also try changing client engines just to see if the issue goes away.

[MrPowerGamerBR]

@e5l after a lot of investigation I found out that it wasn't ktor's issue, it was due to two unrelated issues: My webserver is sometimes getting stuck/taking waaaay too long while compiling Kotlin Scripts (I need to investigate that later) so a lot of requets were getting stuck; The reason it was "queueing requests" and "the timeout only starts after the request was executed" was because I was using GlobalScope without a custom dispatcher, so sometimes the default GlobalScope coroutine dispatcher was full and coroutines start being queued. (and that explains the "queueing requests"-like behavior) 😭

Anyway, thanks for helping 🙂

[e5l]

No problem, I'm happy that the problem is solved 🙂