https://kotlinlang.org logo
Title
q

Quy D X Nguyen

06/08/2020, 9:25 PM
So a bug I've found is that
ApacheClient
will just stop querying large files (3Mb) and not close the connection. This leads to thread starvation and eventually an unresponsive app.
val client = ApacheClient(responseBodyMode = BodyMode.Stream, client = HttpClients.custom()
    .setDefaultRequestConfig(RequestConfig.custom()
        .setCookieSpec(CookieSpecs.IGNORE_COOKIES)
        .setConnectTimeout(3000)
        .setSocketTimeout(3000)
        .build())
    .setMaxConnTotal(10)
    .build())
I'm just piping the resulting stream directly into a response, and when I augmented the ReadCalls with some printlns, they just suddenly stop after a while - and then my app freezes. Edit: It can also happen on smaller files. What's important is that the connection doesn't appear to be closed.
d

dave

06/09/2020, 4:10 AM
hey. could you isolate this into a test case so.we can take a.oser look?
q

Quy D X Nguyen

06/09/2020, 5:27 AM
It appears that sometimes http4k isn't closing the streams that are passed to it in a Repsone. Any ideas what could cause this? Reproducers have not managed to reproduce the bug, were still working on it.
d

dave

06/09/2020, 6:20 AM
in a proxy scenario the only places which even have anything to do with streams (other than passing the reference) are the edges - ie the http client and the ServerConfig layer.
which server are you using? if it's Netty then we would recommend switching it out for undertow, as there have been problems with the benchmarks that we have never managed to reproduce or actually used that backend in prod ourselves.
there is an open issue for that problem which is possibly related to what you're seeing: https://github.com/http4k/http4k/issues/141
s

s4nchez

06/09/2020, 8:14 AM
@Quy D X Nguyen if you don’t consume the response body from the server it may keep the client thread indefinitely, which would explain your issue. I’d look for any conditions (exception or business logic) that would cause you to not read the response body, even for a subset of your requests.
👍 1
q

Quy D X Nguyen

06/10/2020, 4:37 AM
It turns out that the default max connections for apacheclient is 2, and long lasting downloads along with a large number of incoming connections causes Netty to deadlock.
It appears that under load with a blocking handler, the Netty server queues a lot of incoming connections, and then just stops responding. This happens even when streaming, which makes sense given the synchronous nature of http4k.