Is there any way to run the tests such that new spec instance is created for every container that contains tests, but all the tests inside are executed on that one instance? My motivation is that the assertions shouldn’t do any side effects, and so it’s a bit wasteful to repeat the entire setup for each test. So, I’d like this:

DescribeSpec({
  describe("a") {
    println("a")

    describe("b") {
      println("b")

      it("check 1") { println("1") }
      it("check 2") { println("2") }
    }

    describe("c") {
      println("c")

      it("check 1") { println("1") }
      it("check 2") { println("2") }
    }
  }
})

to print

a // first instance
b
1
2
a // second instance
c
1
2

You want InstancePerRoot or something basically ?

Not sure how to call it, really. But yeah, if you define

root

as

container which has at least one test

then exactly

so InstancePerContainer

Tbh I’m not very familiar with the naming, that’s why I went the example route 😄 When I say test it’s the

it() {}

in describe spec, and

describe/context

are containers

there's no way to know if something has assertions until it runs, so it would have to go by the "type" of container

Is it something I can prototype locally somehow, to see the impact?

not without a custom build of kotest

I’ll try that

hmmm

Oookay wait, I have some trouble forcing the local dependency 😕

I would just delete that module locally

I think I managed to publish everything needed, but now when I try to run the test with modified

InstancePerTest

runner I get bunch of

*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message transform method call failed at ./src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 873

😬

what about

DescribeSpec({
  describe("a") {
...
    describe("b", isolate:false) {
      println("b")

      it("check 1") { println("1") }
      it("check 2") { println("2") }
    }
...
})

So, just for context, the reason I asked if it’s easy to prototype was that our tests suite is starting to take a long time, and I was looking for some relatively low-hanging fruit to optimize it. So while

isoloate = false

makes sense, especially if there are technical difficulties without it, ideally I’d like the isolation behavior default for all tests. That said, I still haven’t determined if it’s really worth it 🙂

how long does your suite take?

~1min per module for some biggest modules, and 15 minutes in total, that’s on CI

one possible problem with such a lifecycle is that its not really obvious.why can’t you do singleInstance?

one possible problem with such a lifecycle is that its not really obvious

that’s for sure, that’s why I wanted to measure first. Although arguably

InstancePerTest

is not obvious either, at least with specs that have dedicated assertions block

why can’t you do singleInstance?

Isolating tests is important to me, honestly I don’t even know how to write tests with single instance config, it’d null the biggest benefit of tree-like test structure Kotest offers

Or you can become a kotest contributor and help improve the best test framework on the jvm ;) @christophsturm

cpu I think, we don’t read anything from i/o

And you've enabled parallel execution?

What is proper API for that now?

override val parallelism = 4

in project config class?

Yep that works

Or

concurrentTests/Specs

? Although I already know some of our tests aren’t ready for parallel execution yet due to shared static state in some dependencies. Seems like a noticeable improvement though

Fair

Or you can become a kotest contributor and help improve the best test framework on the jvm ;) @christophsturm 

I’m really hoping that failfast will become one of the (two?) best test frameworks on the jvm 🙂 it may look similar to kotest but i think its absolutely not. there are no features to enable there are no config options there are no different dsls, its for people who mostly care about test speed and think that multi threading is the way to go.

And anything with shared state just put @Isolate on the class

for example I would be really hesitant to add something like @isolate

And parallelism = 4 only parallelizes at the spec level. All tests per spec still run sequentially

And if I have

InstancePerLeaf

? Then a spec is a test anyway, so no difference?

That just controls the instances used not parallelism.

how is @isolate implemented? do you just run the @isolate tests last single threaded? if i would need that i would just put those tests into a separate suite

Yes that's exactly what happens

I forgot a bit of important context: I did log GC for our tests suite, and it looks like each test process triggers GC every ~second, and cleans up roughly 100MB of young objects at every GC. Admittedly I don’t know if it’s a lot or not, but I wanted to see if I can get this number down by limiting Kotest allocations. Since we have a number of tests with many `it`s at the same level, that seemed like an easy target to just experiment.

another example: in failfast there is only instancePerLeaf as lifecycle mode. I have now ported all my test suites to failfast and i keep wanting to use different lifecycle modes but I’m not implementing it yet because I havent found a solution that i like yet. I now think

context("does stuff", isolation: false) {…}

may be a good solution

@wasyl if you find some excessive gc in kotest then let me know and we'll try to fix it. Wasn't there some sourceref thing that was slow that you found a while back.

what happens when you give the test suite more ram? you could probably run the whole suite without doing gc

@christophsturm I’m planning to do that too, I’m gathering more data to know how to tune GC most efficiently 🙂 but with Gradle + parallel tests I’m not sure entire suite without GC is possible @sam I did find sourceref thingy, and yeah when I have something solid I’ll definitely let you know

is @isolate mostly to migrate not well written tests, or do you know usecases where improving the test is not an option and it just needs @isolate ?

If you have shared static state. I use it to test parts of kotest itself.

what shared state does kotest depend on?

If you want solid example — Android’s `ViewModel`s and basically all of Jetpack libraries use shared coroutine dispatchers, so replacing them with test dispatcher to control time can’t be done individually (other than by circumventing the built-in mechanism completely)

interesting. I use kotlin only on server side jvm code. i thought one of the main points of jetpack / swiftui etc is to make ui code better to test.

I don’t, but personal opinion Google doesn’t pay a lot of attention to testing. Tooling is regularly broken, Robolectric is an abomination, APIs aren’t that testing-friendly (with exceptions). The APIs cater to the masses, and let’s be honest, a lot of devs don’t write tests at all, and when they do, they don’t even think about things like parallelism. Case in point about Google’s approach: https://issuetracker.google.com/issues/134473339 open for almost 2 years now, with little activity

i would love to work on an android project and try to find a good way to test it.

Or this one: https://github.com/android/android-test/issues/759, new test APIs behave completely different from real ones, no activity 🤷 This is for instrumented tests but still representative of Google’s stance

yeah thats always the best way to test

So you use kotest for the non android bits

But this is not what Google advocates. They advocate for using their libraries and testing using Robolectric. And they write all the Google libraries with that philosophy in mind

Is robo their library or third party

a lot of kotlin libraries that come from android are also awful.

So you use kotest for the non android bits

we use it for everything that doesn’t require robolectric, so some Android components still qualify, albeit with custom listener to set this global dispatcher state

Yeah the internals of robo are very junit focused

One this issue is done: https://github.com/junit-team/junit5/issues/201 and Robolectric starts using it (probably 3 more years 😛~)~ it’ll get better robolectric might not need to add support

classloader isolation. that looks very interesting

8gb wow

profiling memory usage per test could also be a nice feature