Hei Ktor team. - We have seen this issue since 1....

# ktor

[Satyam Agarwal]

Hei Ktor team.

• We have seen this issue since 1.6.x, and have been bitten by it pretty much every other day in production.
• This issue only occurs on apis where response header has transfer-enconding=chunked and without content length specified.
• As soon as this happens, it poisons the whole ktor client and only way it comes back to normal is by restarting our servers.
• We have seen this issue being raised in youtrack and marked as solved, but we still see it. we are using 2.2.2 as current ktor client version, with CIO as http client engine.
• We have tried removing different http client plugins to eliminate points of failure, but it keeps happening.
• We have also tried using Apache engine in the ktor HttpClient, it still happens.
Problem is, we have never been successful in reproducing this issue locally.
Can you look at the stack trace and give us some clue, eventually, identify if something needs to be done in the ktor client libraries ?

java.io.EOFException: Invalid chunk: content block of size <8192| multiple of 8> ended unexpectedly
	at io.ktor.http.cio.ChunkedTransferEncodingKt.decodeChunked(ChunkedTransferEncoding.kt:93)
	at io.ktor.http.cio.ChunkedTransferEncodingKt$decodeChunked$3.invokeSuspend(ChunkedTransferEncoding.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.internal.LimitedDispatcher.run(LimitedDispatcher.kt:42)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:95)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664)
	Suppressed: java.io.EOFException: Invalid chunk: content block of size 8192 ended unexpectedly
		at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
		at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
		at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
		at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
		at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
		at io.ktor.utils.io.ExceptionUtilsJvmKt$createConstructor$$inlined$safeCtor$3.invoke(ExceptionUtilsJvm.kt:103)
		at io.ktor.utils.io.ExceptionUtilsJvmKt$createConstructor$$inlined$safeCtor$3.invoke(ExceptionUtilsJvm.kt:90)
		at io.ktor.utils.io.ExceptionUtilsJvmKt.tryCopyException(ExceptionUtilsJvm.kt:66)
		at io.ktor.utils.io.ByteBufferChannelKt.rethrowClosed(ByteBufferChannel.kt:2400)
		at io.ktor.utils.io.ByteBufferChannelKt.access$rethrowClosed(ByteBufferChannel.kt:1)
		at io.ktor.utils.io.ByteBufferChannel.setupStateForRead(ByteBufferChannel.kt:295)
		at io.ktor.utils.io.ByteBufferChannel.copyDirect$ktor_io(ByteBufferChannel.kt:3377)
		at io.ktor.utils.io.ByteBufferChannel$copyDirect$1.invokeSuspend(ByteBufferChannel.kt)
		... 8 more
	Caused by: [CIRCULAR REFERENCE: java.io.EOFException: Invalid chunk: content block of size 8192 ended unexpectedly]

Another stacktrace I see on the same issue

java.io.EOFException: Invalid chunk: content block of size 8192 ended unexpectedly
	at io.ktor.http.cio.ChunkedTransferEncodingKt.decodeChunked(ChunkedTransferEncoding.kt:93)
	at io.ktor.http.cio.ChunkedTransferEncodingKt$decodeChunked$3.invokeSuspend(ChunkedTransferEncoding.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.internal.LimitedDispatcher.run(LimitedDispatcher.kt:42)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:95)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664)

[Satyam Agarwal]

@Aleksei Tirman [JB]

[Satyam Agarwal]

Reference to last time I asked this https://kotlinlang.slack.com/archives/C0A974TJ9/p1651051677377989

[Sam]

👀 Watching this thread keenly as we have seen a similar issue and never been able to reproduce it or track it down to a root cause

[Sam]

@Satyam Agarwal do you have an example of the stacktrace when using Apache? So I can compare against the issue I’ve been seeing. In our application the symptom is normally org.apache.http.TruncatedChunkException.

[Satyam Agarwal]

Our logs get purged, but as far as I remember, the stacktrace was very consistent with the one posted above (excluding suppressed exception)

Exception always come from the ktor io utils library, so thats why I wasnt surprised that both Apache and CIO engine suffered from the same issue

[Sam]

Perhaps it’s not the same as my issue 😞

[Satyam Agarwal]

@Aleksei Tirman [JB] @e5l Can you please help ? 😊

[e5l]

Could you log an issue in YT? We can take a look. There is a similar issue recently fixed in the 2.2.3 release. So it would be great if you check 2.2.4 as well