Hello all, I was wondering if anyone else has expe...

# ktor

[Nikola]

Hello all, I was wondering if anyone else has experienced issues with ktor 3.0.2 in tests?

It's regarding a ktor server with ktor tests, where some tests randomly fail with this exception: java.io.IOException: Channel was cancelled with an unclear stack trace that points to either the http client used for the test or the testApplication of the server itself. We observe such failures only on Github CI (Github actions, using a github hosted runner on ubuntu:latest). Reverting to 3.0.0 seems to eliminate these failures (according to our observations so far).

[Nikola]

For anyone who sees this, likely this issue: https://youtrack.jetbrains.com/issue/KTOR-7941

[Erik]

We are experiencing this issue in several of our apps, also with Ktor 3.0.3 (that should have fixed KTOR-7941). I haven't managed to reproduce the exception consistently in a smaller test case. We also only see the issue in Github actions and never locally. Our code is open source and partially written in Norwegian, so this test is perhaps difficult to read, but it is one of the tests that sometimes exhibit the issue:
https://github.com/navikt/hm-soknadsbehandling-db/blob/main/app/src/test/kotlin/no/nav/hjelpemidler/soknad/db/TokenXRoutesTest.kt
I don't think we do anything out of the ordinary, except maybe using Testcontainers for running the required PostgreSQL database.
Did you manage to figure this out @Nikola? Are you still experiencing the issue?

[Nikola]

I also use TestContainers for a pair of Postgres and Redis instances during these tests. I did revert back to ktor 3.0.0 for now, haven't tried 3.0.3 yet. Unfortunately I don't have an opensource project to show you the setup. I will try with 3.0.3 today and will let you know how it goes

[Nikola]

Reverting to 3.0.0 has helped in my case though for sure

[Erik]

Nice! Thank you! I will also try Ktor 3.0.0 to check if it resolves our issue.

[Nikola]

3.0.0 seems to be perfectly stable on my end and I haven't encountered anything that requires me to push to the new patched versions (based on the features and plugins I use) so it's an option for the time being

[Erik]

I was thinking it could be some kind of race condition since it never fails on my Apple MacBook M1 Pro, but sometimes on the Ubuntu VMs in Github CI. Maybe they have less resources than my local setup.

[Erik]

But it is difficult to debug since it only happens on CI.

[Nikola]

Same here, locally on an M3 max it never happens, only on CI, not sure if it's related to resources or the kernel itself somehow?

[Erik]

Not sure, if it is a difference between JVM on Darwin vs Linux then it is above my pay grade 😅

[Erik]

Not sure if this helps anyone, but this is the kind of stack trace we sometimes get in Github CI:

stack_trace_ktor_3.0.3.txt

[Nikola]

@Erik sorry for the delay, but I can confirm that we also still see the issue on ktor 3.0.3...

[Erik]

No problem 😄.

I wonder what we could do to help resolve this issue. I have tried to reproduce the issue here, but no exceptions seen there so far: https://github.com/di0nys1us/ktor-test/blob/main/app/src/test/kotlin/ktor/test/MainTest.kt
I've used @RepeatedTest in an attempt to make the likelihood higher, but still no luck. Maybe something is different compared to our actual applications?

Just let me know if you want to make some changes in the repo, I can add you as a collaborator.

[Erik]

It finally happened! https://github.com/di0nys1us/ktor-test/actions/runs/12633466551/job/35199132365
I will try to create a ticket using this example.

[Erik]

https://youtrack.jetbrains.com/issue/KTOR-8016/testApplication-sometimes-fails-with-java.io.IOException-Channel-was-cancelled

[Erik]

Adding: disable(JsonParser.Feature.AUTO_CLOSE_SOURCE) to our Jackson ObjectMapper fixed the issue for us

[Nikola]

hmm so jackson is the culprit here

[Nikola]

we use it as well

[Nikola]

@Erik do you disable this for the server or the client used for tests? Or both?

[Erik]

Both, but I suspect the server one is the one to blame. I have not tried to disable only the server one though. We usually try to have a single ObjectMapper instance for everything.

[Erik]

Or maybe it is the client, not sure actually

[Nikola]

based on what I implemented it's the server

[Nikola]

I disabled the feature only on the server and things look good on our side now with ktor 3.0.3

[Erik]

Wow, that's great!

[Erik]

Thank you for sorting out this issue with me!

[Nikola]

Likewise, thanks for the effort to get a solution 😁

[Erik]

No problem 😄