Does anybody have some experience working with parallel test execution? Basically we would like to e...

# kotest

[reevn]

Does anybody have some experience working with parallel test execution? Basically we would like to execute multiple specs in parallel to make use of all the cores we have, but we have some test files that do some static mocking, so these files would influence other test files that would run at the same time on a different thread.

Is there any way to leverage the tagging system to mark these files to be run sequentially, while all the other tests can make use of parallelism? If not, do you think this might be something worth exploring in the project?

[sam]

Yep, mark your spec with @DoNotParalleize

[sam]

and they will be sequentially after the rest

[reevn]

Very cool. Will try it out! 👍

[sam]

Let me know how it goes

[reevn]

Hmm. I've overridden parallelism = 4 in our Kotest project config and annotated the ~6 affected test files with DoNotParallelize.

All the tests do indeed pass, but test execution time was about the same (even longer than before).

Are there any caveats to using this feature when using the gradle task to execute tests? From the run window it doesn't look like multiple specs are running at the same time, but that may be a limitation of Gradle?

Also worth mentioning a bit more than half of our tests are still JUnit 4, so in the grand scheme of things most things still are not parallelizable, but I think a difference should definitely be felt if everything works as expected.

[reevn]

Also worth mentioning that we've created our own abstract Spec class that extends DescribeSpec as we needed to override some things for our codebase until we have the new scoped callbacks in place.

[sam]

parallelism would create a thread pool of 4

[sam]

and then it submits all spec tasks into that

[sam]

so it should be running in parallel no problem

[reevn]

I see. When I don't add the annotation, the tests that do static mocking fail when I use parallelism 4, so it has to be working in some way at least. There just is no visual feedback in the run window that multiple specs are running and the rate at which tests pass is also pretty much the same as with one thread.

I haven't had much time to look into this yet, so I will spend some more time down the line figuring this stuff out.

Thanks again for your help!

[sam]

The time should definitely be faster you'd think, perhaps I can write a test with some Thread sleeps to prove it out

[sam]

The visual feedback is just a limitation of gradle's very poor junit5 support

[reevn]

The visual feedback is just a limitation of gradle's very poor junit5 support

Thought so 👍

The time should definitely be faster you'd think

Yeah I'm not sure why I don't see any improvement. I will probably also try some stuff out on my side with some logs when I get the time

[wasyl]

fwiw I’m also seeing minimal improvement (at most 10%) when going from no parallelism to 4x parallelism. Not sure how much improvement is expected, but these are small, under 1 minute suites anyway. Similarly, parallelism does work because some tests using static fields to fail

[sam]

If you have four cores and you're not blocking, I'd expect to see 4x speedup

[wasyl]

I have 6 cores and all are free — nothing running in background except from idle software

[sam]

File an issue if you don't mind and I will investigate later

[wasyl]

I’m also observing a significant delay before the tests start: when running Gradle at first I see Executing tests [x seconds], and only at ~9 seconds it shows first Executing test <class> output.

[wasyl]

Will do that tomorrow 🙂

[sam]

Other people have reported this, my best guess is that the class path scanner is taking a long time on your machine

[sam]

I'd love to be able to reproduce to narrow it down

[wasyl]

I wanted to investigate the delay too. Based on our initial observations there’s some performance regression between 3.4 and 4.x, but it’s difficult to tell for sure — we track test suite execution time, but with caching we rarely execute all tests in all modules.

[wasyl]

I can try profiling/debugging it somehow, although I notoriously fail at trying to profile processes forked by Gradle 😕

[sam]

4.x added in classpath scanning for @AutoScan

[sam]

easy enough to add an option to disable that

[sam]

Actually already an option

[sam]

If you have project config you can turn it off

[wasyl]

I was active in one of the issues to enable it actually, if I didn’t open one 😄 It helps us tremendously by allowing us to plug in listeners from common modules by just adding a dependency

[sam]

try turning it off just to see if the delay goes away

[sam]

then at least we know that's the cause

[wasyl]

Yep, did that just now. So I set autoScanEnabled = false, but I still see the same ~9 seconds delay

[sam]

ok

[wasyl]

I’m logging off for now but I’ll happily investigate more if you have some ideas 🙂

[sam]

Ok I will have a think