Posting this question again, hoping someone will n...

# ktor

[Michael Burbidge]

Posting this question again, hoping someone will notice and answer:

I’m working with push based HTTP API that uses chunked transfer encoding to push messages. I need access to the individual HTTP chunks while reading the response. Is there a way using the ktor HttpClient to get access to the HTTP Chunks?

[e5l]

Hi @Michael Burbidge, what do you mean by chunks(chunks from chunked transfer encoding or stream liked pieces)?

[Michael Burbidge]

I’m making calls against a in-house messaging system similar to Kafka. The client API for consuming messages uses chunked transfer encoding. Essentially the client makes an HTTP request, and the connection stays open. The service continues sending messages in HTTP chunks as response to the request. Messages never span chunks. I’m currently reading the data as a ByteReadChannel and calling readAvailable, but the bytes that I get back from readAvailable span the underlying http chunks and so I have to try and piece the messages back together from the previous call to readAvailable. It would be much easier if I could just read an chunk at a time as the chunks came back over the connection, because a single message is contained within a chunk.

[e5l]

Unfortunately, we have no API to use chunks directly. Moreover, most of the underlying engines(except CIO) don't provide such API 😞 If you have the delimeter in your data, you can use ByteReadChannel.readUntilDelimiter to avoid splitting them by hand

[Michael Burbidge]

Great thanks. I do have a delimiter. So I can try that. Thanks! I hadn’t noticed that call.

[eenriquelopez]

Hi @e5l, is this still current?

I am able to read chunk content, but just as a whole (not accessing each individual chunk). Ideally ktor should be able to access each individual chunk of data, but I am not sure if this is actually supported.

[e5l]

Hey @eenriquelopez, could you tell me why do you need to read individual chunks?

[eenriquelopez]

The use case could be an endpoint that delivers data as it generates it internally (for instance, JSON including different objects)

[eenriquelopez]

I found out this example that seems to be functional: https://github.com/ktorio/ktor-documentation/tree/main/codeSnippets/snippets/client-download-streaming

[e5l]

Yep, you will receive all flushed data

[e5l]

Please note it could be splitted in chunks of different size than server flush it