We are noticing some strange-looking errors from K...

# ktor

[Jeff Hudson]

We are noticing some strange-looking errors from Ktor 3.0, and I’m not sure where to start to debug this.

Request failed. java.lang.IllegalStateException: Check failed.
	at kotlinx.io.Segment.compact$kotlinx_io_core(Segment.kt:234) ~[server.jar:?]
	at kotlinx.io.Buffer.write(Buffer.kt:739) ~[server.jar:?]
	at kotlinx.io.Buffer.transferTo(Buffer.kt:335) ~[server.jar:?]
	at io.ktor.utils.io.ByteChannel.moveFlushToReadBuffer(ByteChannel.kt:79) ~[server.jar:?]
	at io.ktor.utils.io.ByteChannel.awaitContent(ByteChannel.kt:72) ~[server.jar:?]
	at io.ktor.utils.io.ByteChannel$awaitContent$1.invokeSuspend(ByteChannel.kt) ~[server.jar:?]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [server.jar:?]
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:101) [server.jar:?]
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:113) [server.jar:?]
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89) [server.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:589) [server.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:823) [server.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:720) [server.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:707) [server.jar:?]

Wondering if anyone has any ideas…

It does not seem deterministic — this will happen for a given request less than 0.1% of the time, and it will not happen on a subsequent identical request.

[Jeff Hudson]

There’s no cause, and 100% of the stack trace is from Ktor or Kotlin (no references at all to my application’s code)

[Jeff Hudson]

I’m on the latest Ktor 3, latest Kotlin 2, latest everything 🙂

[Jeff Hudson]

image.png

[Filipp Zhinkin]

Hey @Jeff Hudson! Do you have a reproducer that you can share?

[Filipp Zhinkin]

Looking at how the control reaches the failed requirement, successor could only be null (compact's receiver is popped itself, and pop sets both prev and next to null).
So it looks like an issue with concurrent buffer modification (at some other place in the application, not where the exception was thrown).

[Jeff Hudson]

Do you know what these buffers might be used for? It’s a pretty simple CRUD app. I’m not sure we’re manually using this buffer class. I think it’s only used by Ktor under the hood?

[Jeff Hudson]

I can make a reproducer on Monday 🙂 thanks for the look Filipp

[Filipp Zhinkin]

Do you know what these buffers might be used for?

Hard to say (for me), but the reproducer should help

[Bruce Hamilton]

This seems to be the offending function in Ktor https://github.com/ktorio/ktor/blob/52e20f61095bd78ae10c45db268ebbdb971856ae/ktor-io/common/src/io/ktor/utils/io/ByteChannel.kt#L77

So it's likely two coroutines trying to access the read buffer at the same time. I'll log a bug on Ktor for now, since it should at the very least throw a better exception.

[Bruce Hamilton]

I'd be curious if there are any other random exceptions appearing. I'd expect a ConcurrentIOException somewhere.