Nested test suites and testFixture DSL Hi, I was ...

# testballoon

[Astronaut4449]

Nested test suites and testFixture DSL

Hi, I was attending the talk at Kotlinconf and I am excited about this project. I come from Kotest and these are the major selling points, which let me think about migrating our code base to TestBalloon:
• Simplicity
• Extensibility
• Speed through parallel execution
• Clean separation between test discovery and test execution phase
• Recently many regressions in Kotest
Yet, I struggle to migrate our existing tests from Kotest to TestBalloon. Our team relies on a lot of nested tests with a lot of branching. I do not know how to use the fixture DSL in these tests. The fixture is only available in the leaves of the tests, thus there is no way to further modify it while nesting tests.

E.g. given the following test suite:

numbersTest
└── Given an empty list of numbers
    ├── when adding one
    │   ├── the sum should be one
    │   ├── the size should be one
    │   └── when adding another one
    │       ├── the sum should be two
    │       └── the size should be two
    └── when leaving empty
        └── the sum should be 0

Once I define the my list of numbers with

val numbers = testFixture { mutableListOf<Int>() } closeWith { clear() }

There is no way to to obtain and modify it in a nested testsuite.

I found this ugly workaround:

testSuite("Given an empty list of numbers") {
        val numbers = testFixture { mutableListOf<Int>() } closeWith { clear() }

        testSuite("when adding one") {
            val numbers = testFixture { numbers().also { it.add(1) } }

Any suggestion or recommendation?

[Oliver.O]

Happy to hear that you like the project!

The issue in this case is using mutable state at the boundary of blue code (suites) and green code (fixtures, tests). Kotest mixes discovery with execution, which makes your above example easy to describe, but comes with a lot of trouble later on ("when does this before/after thing run?", hence the deprecated isolation modes in Kotest).

TestBalloon has that clear separation, which is comprehensible and free of surprises, but requires a stricter way to describe tests involving shared mutable state.

Your approach is a valid solution. As you have correctly shown, suite-level fixtures can depend on other suite-level fixtures. Just the closeWith() isn't necessary as it will only happen if the list goes out of scope.

I have another solution:

package com.example

import de.infix.testBalloon.framework.core.Test
import de.infix.testBalloon.framework.core.TestFixture
import de.infix.testBalloon.framework.core.TestSuiteScope
import de.infix.testBalloon.framework.core.testSuite
import kotlin.test.assertEquals

val NumbersTest by testSuite {
    testSuite("Given an empty list of numbers") {
        mutableNumbersSuite("when adding one") {
            test("the sum should be one") { numbers ->
                numbers.add(1)
                assertEquals(1, numbers.sum())
            }
            test("the size should be one") { numbers ->
                assertEquals(1, numbers.size)
            }

            testSuite("when adding another one") {
                test("the sum should be two") { numbers ->
                    numbers.add(1)
                    assertEquals(2, numbers.sum())
                }
                test("the size should be two") { numbers ->
                    assertEquals(2, numbers.size)
                }
            }
        }

        mutableNumbersSuite("when leaving empty") {
            test("the sum should be 0") { numbers ->
                assertEquals(0, numbers.sum())
            }
        }
    }
}

private fun TestSuiteScope.mutableNumbersSuite(
    name: String,
    content: TestFixture.Scope<suspend Test.ExecutionScope.(value: MutableList<Int>) -> Unit>.() -> Unit
) = testSuite(name) {
    testFixture {
        mutableListOf<Int>()
    } asParameterForAll {
        content()
    }
}

You could experiment more along these lines, e.g. I've contemplated providing an additional lambda parameter to mutableNumbersSuite for the initial mutation, but you're also mutating state further down ("when adding another one"), which would not be covered.

Maybe you have even better ideas. In this case, I'd like to hear about them.

Where this isn't practical, you can of course always keep your Kotest tests and just use TestBalloon where it is a better fit.

[Oliver.O]

But if you want to do classic Behavior/Gherkin style, you might want to look into
• this use site example, and
• this implementation example.

[Bernd Prünster]

My two cents, trying to model it as close to your request as possible, based on our TestBalloon Addons:

val numberTest by testSuite(testConfig = TestConfig.singleThreaded()) {
    withFixtureGenerator { mutableListOf<Int>() } - {
        testSuite("Given an empty list of numbers") { numbers ->
            testSuite("when adding one") {
                numbers += 1
                test("the sum should be one") {
                    numbers.sum() shouldBe 1
                }
                test("the size should be one") {
                    numbers shouldHaveSize 1
                }

                //copy the numbers list from the outer "when adding one" suite" into a fresh fixture for nesting
                withFixtureGenerator { numbers.toMutableList() } - {
                    testSuite("when adding another one") { numbers ->
                        numbers += 1

                        test("the sum should be two") {
                            numbers.sum() shouldBe 2
                        }
                        test("the size should be two") {
                            numbers shouldHaveSize 2
                        }
                    }
                }
            }

            //fresh fixture: numbers is just  mutableListOf<Int>() from the very top
            testSuite("when leaving empty") { numbers ->
                test("the sum should be 0") {
                    numbers.sum() shouldBe 0
                }
            }

        }
    }
}

BUT: I've just discovered a bug in our Addons' fixture generator and the new versions with the fix won't hit maven Central until Monday

[Bernd Prünster]

Aaaaand the release is out: https://kotlinlang.slack.com/archives/C09FQGG85EC/p1784529141401439.
Just for context: We're using the Addons all across our production stack and we do EAP testing, so the Addons are and will be actively maintained, because we depend on them