hi guys, I wonder what battle tested http client a...

# http4k

[xuemin.guan]

hi guys, I wonder what battle tested http client anyone can recommend to use in our http4k app. We started off with OkHttp. One team reported problems: 1. The default org.http4k.client.OkHttp() use default OkHttp configuration and it seems it is optimized for single user application. 2. the team moved to creating OkHttp with their customized connection pooling configuration but it does not seem to pass their performance testing in some cases.

The team has therefor moved away from OkHttp and started using org.eclipse.jetty.client.HttpClient.

We thought to use org.eclipse.jetty.client.HttpClient instead as well given their experience. But thought to seek some recommendation here.

[dave]

TBH, we've always used OkHttp as the default and never had any problems with it for load, but then again neither have we done any specific testing. I suspect there are some benchmarks somewhere on the web which might be able to help - the http4k shim is pretty slim around the implementations but it would be interesting to know if your team also did any "raw" testing as well to see if there's anything in the shim that might be degrading the perf.

[xuemin.guan]

No one has done “raw” testing. I was not personally involved in the perf testing. This was reported by another team. It’s an issue around OkHttp connection pooling.

They exposed the OkHttp connection pool metrics and during perf tests they observed 1. the pool size was not corrected created 2. the connection was not correctly utilized under loads. 3. it didn’t pass their perf tests because of the connection pool issue.

They then thought it was because they were using the default configuration and OkHttp api doc says it was optimized for a single user app with max 5 connection. They configured the connection pool instead of using the default one by org.http4k.client.OkHttp(). But it still failed their perf test from time to time.

Eventually, they switched to org.eclipse.jetty.client.HttpClient and have never had issue in their perf testing ever since. Their perf test have been running around 6 months now.

[xuemin.guan]

TBH, I am surprised to hear the issue. I had never had issue with it.

[s4nchez]

@xuemin.guan I haven't performance tested okhttp extensively. I'm confident that the apache client is rock solid when it comes to performance. With the appropriate connection pool sizing/monitoring it works fine.

It's worth pointing out also that a lot of performance issues are also related to application design. In general, we recommend one client (with its own connection pool) per "target service", and potentially even individual clients for calls that are known to be of a different performance profile (e.g. file upload vs regular API calls).

[xuemin.guan]

Thank you both!

[xuemin.guan]

I will give the apache a try

[dave]

@xuemin.guan there is a Jetty client as well BTW - did you mean that the team is now using a raw jetty client, or the http4k one (see https://www.http4k.org/guide/modules/clients/ )

[xuemin.guan]

They are creating the Jetty client directly with their configuration then wrap it up in Http4k.

[xuemin.guan]

val client = org.eclipse.jetty.client.HttpClient(SslContextFactory.Client())
        client.start()

jettyHttpClient.connectTimeout = config.connectTimeout.toMillis()
        jettyHttpClient.idleTimeout = config.readTimeout.toMillis()
        val jettyHttp4kClient: HttpHandler = org.http4k.client.JettyClient(client = jettyHttpClient)

[xuemin.guan]

something along those lines ☝️

[dave]

that's fine - just checking that they aren't throwing out the abstraction as well!

[Razi Kheir]

@xuemin.guan Where are you defining the connection pool using JettyClient? as our performance tests seemed to be not amazing with OkHttp, at 1000 qps cpu was at 100%.

[Razi Kheir]

Any Idea why Netty would throw

HTTP protocol violation: Authentication challenge without WWW-Authenticate header

[Razi Kheir]

Sometimes it throws 500 although response code is 200 but but reason is null.. any idea how I can disable netty’s internal checks?

[dave]

TBH - we have never used Netty in prod so can't really help. I'd suggest using undertow instead. Its possible that you're hitting the same kind of problems that we have seen in the benchmarks (but haven't reproduced ever locally). Can you put together a simple gist that shows this happening ?

[Razi Kheir]

Hmmm I need to think how to do that without exposing the specific calls 🙂
I’ll try to create a new client / request/ response and check if I can replicate it like that on the week end 🙂

[dave]

If it helps, just the simple shell of the app with the kinds of calls that you’re doing (method/content type/length/lenses etc) and a client calling them in a loop. Anything would help because quite frankly we’ve never seen it! 😆