hi folks, I have a issue with data driven testing....

# kotest

[abendt]

hi folks,
I have a issue with data driven testing. Given this example:

class DataDrivenExampleSpec : StringSpec({

    extension(MyListener())

    withData(listOf(1, 2)) {
        println("Test $it")
    }

    "single 1" {
        println("single 1")
    }

    "single 2" {
        println("single 2")
    }
})

class MyListener : BeforeTestListener, AfterEachListener, AfterTestListener {
    override suspend fun beforeTest(testCase: TestCase) {
        println("beforeTest")
    }

    override suspend fun afterEach(testCase: TestCase, result: TestResult) {
        println("afterEach")
    }

    override suspend fun afterTest(testCase: TestCase, result: TestResult) {
        println("afterTest")
    }
}

i would expect that the two after* callbacks to behave similiar regardless if the test is a data driven one or not.

However i am seeing that afterEach is not invoked for the data driven tests. Is this intended?

beforeTest
Test 1
afterTest
beforeTest
Test 2
afterTest
beforeTest
single 1
afterTest
afterEach
beforeTest
single 2
afterTest
afterEach

[Davio]

I don't think the data tests are not part of a specific test case, they are just run in the root of the test

[Davio]

What happens when you surround it with a context?

[Davio]

context("ctx") { withData ... } }

[Davio]

Also I don't known about the special listener class, maybe using beforeTest { ... } inside of your spec works better?

[abendt]

this is a StringSpec it doesn't support context.

I have created a ShouldSpec based example that shows a similar behaviour:

class DataDrivenExampleShouldSpec : ShouldSpec({

    extension(MyListener())

    context("data") {
        withData(listOf(1, 2)) {
            println("Test $it")
        }
    }

    context("direct") {
        should("single 1") {
            println("single 1")
        }

        should("single 2") {
            println("single 2")
        }
    }
})

output

beforeTest
beforeTest
Test 1
afterTest
beforeTest
Test 2
afterTest
afterTest
beforeTest
beforeTest
single 1
afterTest
afterEach
beforeTest
single 2
afterTest
afterEach
afterTest

[abendt]

About the listener: I disovered this behaviour while trying to use FluentLenium (https://fluentlenium.com/docs/test-runners/#kotest) in combination with data driven tests. The FluentLenium integration internally uses the extension mechanism + listener class to integrate into the lifecycle of the test. This example is just a stripped down example of that.

[Davio]

Could you replace the explicit listener class with just inlining beforeTest in your spec?

[Davio]

Should not matter, but can't hurt to try

[abendt]

Sure:

class DataDrivenExample2Spec : StringSpec() {

    override suspend fun beforeTest(testCase: TestCase) {
        println("beforeTest")
    }

    override suspend fun afterEach(testCase: TestCase, result: TestResult) {
        println("afterEach")
    }

    override suspend fun afterTest(testCase: TestCase, result: TestResult) {
        println("afterTest")
    }

    init {
        withData(listOf(1, 2)) {
            println("Test $it")
        }

        "single 1" {
            println("single 1")
        }

        "single 2" {
            println("single 2")
        }
    }
}

beforeTest
Test 1
afterTest
beforeTest
Test 2
afterTest
beforeTest
single 1
afterTest
afterEach
beforeTest
single 2
afterTest
afterEach

[Davio]

I think I found something

[Davio]

afterEach only works for TestCases of type "Test", and the withData testcases have type "Dynamic"

[Davio]

afterTest runs after any test, so both those with type Test and type Dynamic

[Davio]

It doesn't look like there is a specific function for "after Dynamic test": https://kotest.io/docs/framework/extensions/simple-extensions.html

[Anouar di Kali]

@Davio @abendt it seems like only afterTest and afterAny work with dynamic test

class Demo : StringSpec({

    afterTest {
        println("AfterTest")
    }

    afterAny {
        println("AfterAny")
    }

    withData(listOf(1, 2)) {
        println("Test $it")
    }
})

output:
Test 1
AfterTest
AfterAny
Test 2
AfterTest
AfterAny

[abendt]

thanks @Anouar di Kali and @Davio that was helpful

changing to afterAny works for me. I have made the change in the fluentlenium codebase and tests pass.

Still think its a bit surprising that afterEach is not invoked for dynamic tests. IMO it should behave similarly regardless how the test was defined. wdyt?

[Davio]

I don't know maybe it was designed before dynamic tests existed and it was never considered

[sam]

Yes what Dave said. Dynamic is new. In kotest 6 we won't rely on junit platform anymore and these distinctions can go away.

[Davio]

Would it make sense to get a PR in to have it covered before that time?

[sam]

Yes sure