A modern programming language that makes developers happier.

kotlinlang

Hey all. Maybe you guys can help us out. We are using the ktor apache http client and we get random "Connection reset by peer" exceptions . We only observed this for a particular service that runs behind a load balancer. So the assumption is that if the client gets balanced to another node, it won't have a valid connection anymore.
To start with, I don't understand how http clients are meant to be used in ktor. Our ktor backend acts as a gateway between a frontend and multiple backend services. So the question is: should we recreate a http client instance for every request and then immediately close the client, as it is shown in the ktor examples? <https://ktor.io/clients/http-client.html>
Or should we initialize the client once, say as part of the initialization of an object, and then reuse it for all requests?
What happens when a http client is instantiated? I once forgot to close a client and it seemed on every instance creation it spawned 10 new threads, which indicates to me that it is meant for reuse?

I'd say that you should try to reuse the client as much as possible.
When instantiating an `HttpClient`, it also creates its `HttpClientEngine`, which is `Apache` in your case.
From `HttpClientJvmEngine`, you can see that all JVM engines allocate a pool of `threadsCount` daemon threads to execute requests in parallel, where `threadsCount` can be changed in the `engine { ... }` block (defaults to 4).
Closing an `HttpClient` will at least cancel its pending requests and release its thread pool. Since those are valuable resources, you'd initialize the client once and reuse it as much as possible.

<@UFQKL22HE> thank you for the answer. that's exactly what I was thinking as well.
We were able to "solve" our issue by adding this to the client config:
```
engine {
            customizeClient {
                setConnectionReuseStrategy(NoConnectionReuseStrategy())
            }
        }
```

So there is no connection reuse, and the client always makes calls with a fresh connection that the target could not have hung up.

This seems more like a workaround to me, than a proper solution, though. I got the idea from this thread on github, which is similar but for netty: <https://github.com/reactor/reactor-netty/issues/388>

I also read somewhere else, that a connect reset by peer is a "valid use case" that clients should be prepared to deal with (by retry). So I am not sure whether I should wrap all my calls in a retry handler to deal with the problem (seems a bit weird?), or whether the disable connection reuse thing is better (performance impact?). Right now I am simply using whatever keeps my service stable ...

Reading up on this HTTP keep alive mechanism. It seems the most probably thing happening here is that our server is closing the underlying TCP connection after some time, and the apache client by default assumes the connections stay valid forever (except there is a Keep-Alive header with timeout in the response, which there is not). So probably, instead of disabling the feature altogether, it would be better to set a relatively low timeout within the client already