i have a generative test case that generates over 7k tests W

Question

i have a generative test case that generates over 7k tests When run by itself in intellij just the test class the test completes in 4 seconds If instead I select the directory and run all tests in the whole project the tests take over 22 minutes to complete Each consecutive test takes longer than the one before to complete No clue what is going on here anyone ever seen this

sam · Answer

What do you mean by generative test case

snowe · Answer

here s my exact test case ``` context break should occur between javaClass getResource LineBreakTest txt readText lines forEach nextTest testLine gt if testLine startsWith or testLine isBlank return nextTest val parts = mapUnicodeTestLineToParts testLine val padding = val testName = testLine split 1 if testsToSkipCurrently contains testName return nextTest it testName <http logger info|logger info> testName logger debug parts prettyPrint wrappedLineWidth = 1 obj = LongString parts flatten joinToString mapTo LongString longString = $ $ parts joinToString n$padding it joinToString $ ```

sam · Answer

``` return nextTest```

sam · Answer

where s nextText

sam · Answer

Is this a simplified example because you re not actually doing any assertions in it

snowe · Answer

nextTest is the name of the `forEach` at the top And the assertion is in mapTo

sam · Answer

ok I see now yeah

sam · Answer

and linebreaks txt has about 7k lines

snowe · Answer

yep

sam · Answer

and even if they all pass it takes forever

snowe · Answer

yeah

sam · Answer

sounds like an o n issue in the junit listener

sam · Answer

``` class FooTest FunSpec init List 10000 it forEach test test $it 1 shouldBe 1 ```

sam · Answer

Try that for me

snowe · Answer

you can see the full codebase here here is the multiline string test and here is the linebreak file but note that that s all before the conversion to kotest

sam · Answer

If you try the test case I just pasted Takes 5 seconds for me

sam · Answer

If you run that in gradle I think it will take a long time no need to run it to completion just see if it s lagging

snowe · Answer

takes 10s in intellij already over 30 seconds in gradle and that s only 1500 cases

sam · Answer

yeah its lagging for me in gradle

sam · Answer

so it s almost certainly junit listener

sam · Answer

let me fix and push a snapshot and we can try that

snowe · Answer

oh sweet

snowe · Answer

woah I let that test case run to completion and it ran 99 extra tests

sam · Answer

that s odd

snowe · Answer

it s like it ran an extra test for every 100 tests

sam · Answer

better safe than sorry

snowe · Answer

hahaha

sam · Answer

I ve fixed it Just writing a test

snowe · Answer

awesome let me know when to test on my end

sam · Answer

sam · Answer

Ok give 4 3 0 621 SNAPSHOT a go

sam · Answer

don t forget to add the snapshots maven repo

snowe · Answer

sorry got super busy trying now

snowe · Answer

awesome 3 9s for 7284 tests Thanks so much

snowe · Answer

ah wait I ran it using the intellij runner

snowe · Answer

darn still slow much faster than before though ran about 4k tests in 60 seconds but is slowing down from there 6700 tests in 2 minutes

sam · Answer

you re definitely on the snapshot everywhere

snowe · Answer

yeah let me post my dependencies I can also push my branch if you want to try itout

sam · Answer

ok that would help yeah

sam · Answer

I m sure it will be something simple and I ll be able to fix it quickly

snowe · Answer

correction I had changed the version of the kotest pitest plugin lol after changing the correct version it builds in 37 seconds which is a major improvement It still doesn t run as fast as intellij standalone but that is fine

snowe · Answer

I was distracted when changing the version number and then subsequently responding to you Sorry about that face palm skin tone 4

sam · Answer

the difference between intellij and junit will be due to the way the junit output writer works

sam · Answer

but if it only took 2 seconds in spek and it takes 37 seconds now then that s not good enough

sam · Answer

assuming the tests were identical

snowe · Answer

let me push my branch then and get you the spek version as well I think the spek tests were partially borked on master

snowe · Answer

and yeah I literally just replaced `Spek` with `DescribeSpek` and removed the spek setup function I had

sam · Answer

2 seconds does seem pretty fast for 7k tests

snowe · Answer

yeah it does actually but it runs in 4 seconds on kotest in intellij so maybe not that farfetched

sam · Answer

true

sam · Answer

I get 5 seconds if I run via gradle mind on my dummy loop 10k loop test

sam · Answer

There could be another cause of slowness What s your map to method doing

sam · Answer

well I can take a look when you push

snowe · Answer

just realized I had parallelism set in the kotest config let me get rid of that real quick and see how fast it is

snowe · Answer

actually I think it s the compilation step in intellij that s taking so long I ll still push my branch for you one se

snowe · Answer

snowe · Answer

ok I ran it with just this command and it only took 10s so that s super good ``` gradlew test tests= com tylerthrailkill helpers prettyprint MultilineStringTest gt Configure project Inferred project pretty print version 2 1 0 dev 15 uncommitted+kotest 21f0e7a Deprecated Gradle features were used in this build making it incompatible with Gradle 7 0 Use warning mode all to show the individual deprecation warnings See <https docs gradle org 6 5 1 userguide command line interface html sec command line warnings> BUILD SUCCESSFUL in 10s 6 actionable tasks 2 executed 4 up to date```

snowe · Answer

oh that wasn t the greatest output but yeah you get it it did actually run all the tests

sam · Answer

ok perf

sam · Answer

not much for me to look at then

sam · Answer

Why did you migrate from spek out of interest

snowe · Answer

honestly I tried kotest out and it immediately was a much better experience Migration was the easiest migration I ve ever seen There was only a single dependency to add whereas with spek I had to exclude certain dependencies I also had to create special setup functions for spek that I don t need at all in kotest no clue why I needed memoization in a spek setup function in order to test properly The spek intellij plugin fails all the time and is pretty slow The kotest plugin is incredibly fast and worked perfectly first time That alone was enough for me One thing I didn t like initially was that spek allows me to have duplicate test names kotest does not That 7k line file from before had quite a lot of duplicates that I didn t know about until switching to kotest hahahaha That was a one time fix though but it did take a while because kotest fails on the first duplicate instead of finding all duplicates Also kotest doesn t output the duplicate name I had to debug into kotest and put a watch to see what the name was just to delete the right line from the LineBreakTest txt file Finally you have a PIT plugin Spek is completely incompatible with PIT and I think mutation tests are super powerful That alone would have been enough for me to switch with none of the other benefits Even if the experience had been worse I still would have switched

sam · Answer

Right that s really cool I think kotest is the best test framework but of course I m heavily biased

sam · Answer

As for Also kotest doesn t output the duplicate name good shout I ll check that now and fix it

snowe · Answer

awesome Hopefully that helps someone else further down the road And hopefully me if I ever update that linebreak file joy

snowe · Answer

also I actually love the FreeSpec style It looks so good in nested tests with forAll rows

sam · Answer

I never liked that one so much slightly smiling face

sam · Answer

snowe · Answer

I actually didn t like it when I first saw it as well but then decided to try it out and it s crazy readable in certain contexts I could definitely see how it wouldn t be readable or nice in other contexts

sam · Answer

I was a scala developer for 9 years

sam · Answer

FreeSpec is inspired from <https www scalatest org at a glance FreeSpec>

snowe · Answer

Ah I can definitely see how it would not be great in that context My current team is me and one other dev So I think we will be able to maintain it pretty decently Also we were hired to literally trash the monolith we re working on so all the FreeSpec stuff will be gone eventually lol Maybe I ll figure out if I hate it before then and decide to use a different form What s your favorite

sam · Answer

I wasn t trying to convince you to not use it slightly smiling face I think free spec is very popular because it s good when you want to generate tests like you are

sam · Answer

FunSpec is my favourite because I m old and like to do things without nesting

snowe · Answer

ah haha I think nesting is the most powerful test construct I think it makes stuff way more readable and refactorable I know a lot of people don t like it though

sam · Answer

If I do end up with some nesting for whatever reason I like describe spec

sam · Answer

because it copies the javascript style

snowe · Answer

yeah that was the one I started off using I also code a lot in ruby so used to that as well

sam · Answer

I think it started off with rspec and then the js frameworks borrowed it

snowe · Answer

yeah

snowe · Answer

anyway thanks for the hard work I love your library

LeoColman · Answer

gt FunSpec is my favourite because I m old and like to do things without nesting Particularly I use ShouldSpec for almost every scenario