https://kotlinlang.org logo
Title
w

wasyl

05/19/2023, 11:34 AM
We encountered some weird Kotest behavior and it's unclear to us if it's a bug or we're doing something wrong. Basically using
runTest
and
backgroundScope
, if a job in
backgroundScope
fails, then all tests pass. However, this only happens if there's another parallel test container that has a job in
backgroundScope
that never ends. This is very surprising, as I thought
backgroundScope
would be properly cleaned up, and overall having a dependency like this looks like a bug? Here's repro code:
package com.repro

import io.kotest.core.spec.IsolationMode
import io.kotest.core.spec.style.DescribeSpec
import io.kotest.matchers.shouldBe
import kotlinx.coroutines.flow.firstOrNull
import kotlinx.coroutines.flow.flow
import kotlinx.coroutines.launch
import kotlinx.coroutines.test.UnconfinedTestDispatcher
import kotlinx.coroutines.test.runTest

class Repro : DescribeSpec() {

    override fun isolationMode() = IsolationMode.InstancePerLeaf

    init {
        describe("repro") {
            runTest(UnconfinedTestDispatcher()) {
                describe("tested expression") {
                    backgroundScope.launch { error("Error in background scope") }
                    // coroutineScope { launch { error("Error in non-background scope") } }

                    it("should not be reached?") {
                        Unit
                    }
                }

                describe("background scope here never ends") {
                    val neverEnds = backgroundScope.launch {
                        flow<Unit> { }.firstOrNull()
                    }

                    it("should fail") {
                        neverEnds.isCompleted shouldBe true
                    }
                }
            }
        }
    }
}
Things to try (the expected behavior is for these tests to fail): • running this test as is will pass • making the second
describe
an
xdescribe("background scope here never ends")
will not execute the second test, but tests still passremoving the entire
describe("background scope here never ends")
block makes tests fail
• swapping the describes with each other (changing the order) makes tests fail too • Uncommenting
Error in non-background scope
line will make tests fail • changing the isolation mode to
SingleInstance
will make tests fail We tried to debug this but we're at a loss. I thought
InstancePerLeaf
should ensure that tests are pretty much independent, and I completely don't see where or what could swallow background scope errors, or why background scope would not propagate the errors as we expect. (a note — we're trying to use Coroutines'
runTest
directly because back when Kotest didn't have support for background jobs we just created own abstractions and it felt reasonable to use native APIs when possible, even within Kotest context. We also use the same extensions for non-Kotest tests, so it'd be nice to keep using
runTest
, unless it's inherently incompatible with Kotest)
l

LeoColman

05/19/2023, 2:12 PM
I think you should turn this into an issue. I don't think we'll wrap it up with a quick discussion here xD
w

wasyl

05/19/2023, 2:13 PM
Yeah initially I wasn't sure if it's a bug so figured I'll ask here first. I'll open a proper issue later today 👍