Hey ktor community! I need to understand content c...

# ktor

[Olaf Gottschalk]

Hey ktor community!
I need to understand content compression from the perspective of a Ktor client sending HTTP posts to a server....

From the documentation, I read that I need to install the ContentEncoding plugin to the HTTP client, but also need to explicitly compress the body of a request each time using compress("gzip") for example. But: how can I test that this really does compress the data? Is there any way to see this? When I inspect the HttpResponse object for compression headers after the call, I see none.
Also, I tried to confirm it will do some form of compression by deliberately not installing the plugin but trying to compress the payload - I wanted to provoke some "cannot compress, missing plugin" exception/error that way. But: nothing happens. I can compress in my request builder, even without installing the plugin first. I also installed a server that does not allow compression, but still never saw any error pop up. I can't help but feel, I don't understand this feature.

Then, I set my log level to ALL using install(Logging) { level = LogLevel.ALL } and saw this:

06:52:13.961 [DefaultDispatcher-worker-2 @coroutine#12] INFO  i.k.c.HttpClient - REQUEST: <http://localhost:64532/ws-agathe/xyz>
METHOD: POST
COMMON HEADERS
-> Accept: application/xml
-> Accept-Charset: UTF-8
-> Accept-Encoding: gzip;q=0.9
-> Authorization: Basic Zm9vOmJhcg==
CONTENT HEADERS
-> Content-Length: 133
-> Content-Type: text/xml
BODY Content-Type: text/xml
BODY START
<env:Envelope xmlns:env="<http://www.w3.org/2003/05/soap-envelope>"><env:Header/><env:Body><xml>1 of 10</xml></env:Body></env:Envelope>
BODY END

Somehow, this feels wrong altogether. It accepts gzip, yes, and that is exactly what I configured when installing the ContentEncoding plugin. But: my request body is not compressed at all, even though in the HttpRequestBuilder I do a compress("gzip"). Why?

Even more: other libraries that offer some sort of compression always have some configurable threshold as to when to compress depending on the payload size. In Ktor, would I have to manually do an if (size > threshold) compress("gzip") do achieve the same effect?

Thank you so much for any more insights and probably improvement of documentation afterwards!
Olaf

[Aleksei Tirman [JB]]

There is a mode configuration property that can be used to enable request body compression (Mode.CompressRequest), response body decompression (Mode.DecompressResponse), or both. By default, only the response body decompression is enabled.
The general idea is that you list all the available encoders in the configuration of the plugin, and then in the HttpRequestBuilder , call the compress method to specify which one to apply to the current request body.
You can test that the request body is actually compressed with the following code:

val client = HttpClient(CIO) {
    install(ContentEncoding) {
        mode = Mode.All
        gzip()
    }
}

val response = <http://client.post|client.post>("<https://httpbin.org/post>") {
    setBody("a".repeat(64 * 1024))
    compress("gzip")
}

println(response.bodyAsText())

In the response body, you will see the compressed request body under the data key.

[Olaf Gottschalk]

Thanks a lot! Will try this! So, what I did not do (before) was set this mode, as it is not documented in said page.
And, tbh, I also don't understand the reason why when I explicitly call compress in the HttpRequestBuilder before setting the body, this does compress the request...

[Olaf Gottschalk]

It worked! Setting the mode and calling compress("gzip") is both required. So, why is that piece missing in the documentation? And, another thing... what if the server does not accept compressed posts? How does my client detect the features before trying to send compressed data? Does it automatically fall back to uncompressed on certain response codes? Or do I have to manually program all that logic?

[Aleksei Tirman [JB]]

Can you please file an issue to address the problem with the documentation? If the server does not accept compressed posts, it should respond with 415. Also, it can signal the client via the Accept-Encoding header which encodings it supports. The client cannot detect the server's capabilities before sending the request, but you could know them.

Does it automatically fall back to uncompressed on certain response codes?

No, it does not.