We encountered some weird Kotest behavior and it's...

# kotest

[wasyl]

We encountered some weird Kotest behavior and it's unclear to us if it's a bug or we're doing something wrong. Basically using runTest and backgroundScope, if a job in backgroundScope fails, then all tests pass. However, this only happens if there's another parallel test container that has a job in backgroundScope that never ends. This is very surprising, as I thought backgroundScope would be properly cleaned up, and overall having a dependency like this looks like a bug?

Here's repro code:

package com.repro

import io.kotest.core.spec.IsolationMode
import io.kotest.core.spec.style.DescribeSpec
import io.kotest.matchers.shouldBe
import kotlinx.coroutines.flow.firstOrNull
import kotlinx.coroutines.flow.flow
import kotlinx.coroutines.launch
import kotlinx.coroutines.test.UnconfinedTestDispatcher
import kotlinx.coroutines.test.runTest

class Repro : DescribeSpec() {

    override fun isolationMode() = IsolationMode.InstancePerLeaf

    init {
        describe("repro") {
            runTest(UnconfinedTestDispatcher()) {
                describe("tested expression") {
                    backgroundScope.launch { error("Error in background scope") }
                    // coroutineScope { launch { error("Error in non-background scope") } }

                    it("should not be reached?") {
                        Unit
                    }
                }

                describe("background scope here never ends") {
                    val neverEnds = backgroundScope.launch {
                        flow<Unit> { }.firstOrNull()
                    }

                    it("should fail") {
                        neverEnds.isCompleted shouldBe true
                    }
                }
            }
        }
    }
}

Things to try (the expected behavior is for these tests to fail):
• running this test as is will pass
• making the second describe an xdescribe("background scope here never ends") will not execute the second test, but tests still pass
• removing the entire describe("background scope here never ends") block makes tests fail
• swapping the describes with each other (changing the order) makes tests fail too
• Uncommenting Error in non-background scope line will make tests fail
• changing the isolation mode to SingleInstance will make tests fail
We tried to debug this but we're at a loss. I thought InstancePerLeaf should ensure that tests are pretty much independent, and I completely don't see where or what could swallow background scope errors, or why background scope would not propagate the errors as we expect.

(a note — we're trying to use Coroutines' runTest directly because back when Kotest didn't have support for background jobs we just created own abstractions and it felt reasonable to use native APIs when possible, even within Kotest context. We also use the same extensions for non-Kotest tests, so it'd be nice to keep using runTest, unless it's inherently incompatible with Kotest)

[LeoColman]

I think you should turn this into an issue. I don't think we'll wrap it up with a quick discussion here xD

[wasyl]

Yeah initially I wasn't sure if it's a bug so figured I'll ask here first. I'll open a proper issue later today 👍

[wasyl]

Opened https://github.com/kotest/kotest/issues/3537