Hey all! 1. Do we know when does v6 go live? 2. Ab...

# kotest

[Shervin]

Hey all!
1. Do we know when does v6 go live?
2. About the upcoming changes in Isolation mode, do we expect huge breaking changes? Our app is heavily using 'per leaf' mode. I want to estimate how much of 'tech debt' will be added to our back log
Thanks in advance!

[dirk.dittert]

I don't believe that there was any communication about it. But there is a milestone in Github

[sam]

Still in the air. I may add it back if possible but it's really annoying feature as it creates issues everywhere else. Tbd

[LeoColman]

1. We are releasing snapshots from time to time, but the final version is coming only when we feel 100% comfortable with it

[LeoColman]

And that really has no date. It's unpaid and passionate work that can't be rushed

[Shervin]

1. clear, and thats even better for us! if anything we can use the time to prepare better for the [possible] migration
2. I still need to understand what will be the impact after removal of per leaf isolation mode. Do we need to create multiple test cases (in case that we are using Behaviour spec) instead ?

[dirk.dittert]

I do understand the consequences of using open source libraries and products. From the perspective of user I find it quite hard to prepare for 6.0, though. I am wondering what the major changes will be and how my code will be impacted by that.

[sam]

well before 6.0 comes out I will write a big blog post with the changes

[sam]

I just don't wanna do that yet because I am not sure of the final changeset (I want as few breaking changes as possible)

[Matej Drobnič]

Doesn't removing "per leaf" option break the fundamental aspect of the tests - that all tests should be isolated from each other?

Without that option, When/should/scenario blocks are affected by all the other blocks before it.

[Lukasz Kalnik]

For us as well removing the "per leaf" isolation would break our test concept.
We create the system under test and mocks before all tests, but then run the tests in isolation on them.
I guess another approach would be to use the test setup / teardown callbacks, but you cannot return the system under test and mock instances from them. Probably we will need some global nullable vars then. Doesn't sound very idiomatic in Kotlin.

[Lukasz Kalnik]

I wonder why the "per leaf" isolation doesn't seem to be convincing/popular. Do people not store state in their classes? Do they design their tests so that they are able to run them one after another and they don't break each other, even when there are changes?

[Lukasz Kalnik]

In the Android world when testing a ViewModel it keeps a lot of state. If you add a new test in a "single instance" mode, then you have to keep in mind all the previous interactions with the viewmodel. What if you want to test some scenarios which are disconnected from each other?
Also all the recorded mocks have to be reset before each test (which should be less effort, as you could do it in test lifecycle hooks).

[LeoColman]

Personally, no, I don't store state between tests.
I usually have a full setup for each test and a full tear down for each test as well. That for Unit Tests.

For Integration Tests I usually keep a single state and thus SingleInstance is enough

[LeoColman]

So InstancePerLeaf on my use case would lead to an equal behavior as InstancePerTest

[Lukasz Kalnik]

Do you program backend or Android?

[LeoColman]

Both. About 7~ years each with Kotlin + Kotest

[LeoColman]

I probably write android incorrectly tho. Always indie

[Lukasz Kalnik]

There is no "incorrectly" 😉

[Lukasz Kalnik]

I guess we could also instantiate the Viewmodel in every test. Then it's a bit more overhead, but will still work. And have a setup() method to setup all the mocks, and return them as some list.
Just seems not very clean and a lot of repetitive code.

[Lukasz Kalnik]

I would probably consider going back to JUnit.

[LeoColman]

Hmm... Could you give me an example of what becomes too verbose?

I have a JUnit background as well, and I never even knew there was an InstancePerLeaf on JUnit. I never 'needed' this kind of solution on my use cases, probably

[Lukasz Kalnik]

That's how we do it with InstancePerLeaf:

MyTest : StringSpec({
    val viewModel = MyViewModel()
    val repo = mockk<MyRepo> {
        every { doSomething() } returns true
    }
    val otherRepo = mockk<OtherRepo>()
    
    "when otherRepo.doSomethingElse() returns null then viewModel shows loading state" {
        every { otherRepo.doSomethingElse() } returns 0
        viewModel.refreshData()
        viewModel.uiState.value shouldBe Loading
    }

    "when otherRepo.doSomethingElse() returns -1 then viewModel shows error state" {
        every { otherRepo.doSomethingElse() } returns -1
        viewModel.refreshData()
        viewModel.uiState.value shouldBe Error
    }
}

[Lukasz Kalnik]

When you don't have instance per leaf, you have to do the common initialization in every test.

[LeoColman]

Wouldn't InstacePerTest work the same way with a bit more overhead?

[Lukasz Kalnik]

And additionally reset the viewmodel / mocks after every test

[Lukasz Kalnik]

InstancePerTest is for me the weirdest one.

[LeoColman]

InstancePerTest is the default for JUnit

[Lukasz Kalnik]

How so? Does JUnit have such configuration?

[LeoColman]

No, I mean analogously. The match for InstancePerTest is JUnit

[LeoColman]

While SingleInstance is the default in Scalatest

[LeoColman]

InstancePerLeaf I think is our invention IIRC

[Lukasz Kalnik]

InstancePerTest is for nested tests

[Lukasz Kalnik]

I mean the difference between per leaf and per test is visible with nested tests only.

[Lukasz Kalnik]

E.g. BehaviorSpec or FreeSpec

[Lukasz Kalnik]

Here it wouldn't make any difference

[Lukasz Kalnik]

But in FreeSpec it woul als re-execute every container test

[LeoColman]

I go for InstancePerTest even on nested tests (context/test on FunSpec). The only problem is that it resets my mocks and others more times than I should, but my tests never care about this

[Lukasz Kalnik]

with and without their matching children

[Lukasz Kalnik]

For me the running of the container tests by PerTest without their children doesn't make sense

[LeoColman]

Containers are in theory a Container + Test. As breaking it makes the suite fails

[Lukasz Kalnik]

the tests, when nested, are designed as "scenarios" which branch out into the children (when there are multiple)

[Lukasz Kalnik]

But if you run a leaf you necessarily run the container as well, so you see if it breaks or not

[Lukasz Kalnik]

That's why I don't understand the point of InstancePerTest

[LeoColman]

Yes, but if you ignore the leaf the container still runs

[Lukasz Kalnik]

Hmm ok

[Lukasz Kalnik]

I don't ignore tests ever 😉

[Lukasz Kalnik]

Maybe that's why

[LeoColman]

me neither. I comment them 😂

[Lukasz Kalnik]

That's also a solution 🙂

[LeoColman]

I think the point of instance per test is both having the JUnit option for people that are used to how junit does and also for having a nuclear option "Run from root to leaf uninterrupted"

[Lukasz Kalnik]

We also have a small codebase, only 250 tests