https://kotlinlang.org logo
Join Slack
Powered by
i have a generative test case that generates over ...
# kotest
s
i have a generative test case that generates over 7k tests. When run by itself in intellij (just the test class), the test completes in 4 seconds. If instead, I select the directory and run all tests in the whole project, the tests take over 22 minutes to complete. Each consecutive test takes longer than the one before to complete. No clue what is going on here. anyone ever seen this?
s
What do you mean by generative test case
s
here's my exact test case:
Copy code
context("break should occur between ") {
            javaClass.getResource("/LineBreakTest.txt")
                .readText()
                .lines()
                .forEach nextTest@{ testLine ->
                if (testLine.startsWith('#') or testLine.isBlank()) {
                    return@nextTest
                }
                val parts = mapUnicodeTestLineToParts(testLine)
                val padding = "                                       "
                val testName = testLine.split('#')[1]
                if (testsToSkipCurrently.contains(testName)) {
                    return@nextTest
                }
                it(testName) {
                    <http://logger.info|logger.info> { testName }
                    logger.debug { parts }
                    prettyPrint(
                        wrappedLineWidth = 1,
                        obj = LongString(parts.flatten().joinToString(""))
                    ) mapTo """
                        LongString(
                          longString = ""${'"'}
                                       ${parts.joinToString("\n$padding") { it.joinToString("") }}
                                       ""${'"'}
                        )
                        """
                }
            }
        }
s
Copy code
return@nextTest
where's nextText
Is this a simplified example because you're not actually doing any assertions in it
s
nextTest is the name of the 
forEach
at the top. And the assertion is in mapTo.
s
ok I see now yeah
and linebreaks.txt has about 7k lines
s
yep
s
and even if they all pass it takes forever
s
yeah.
s
sounds like an o(n) issue in the junit listener
Copy code
class FooTest : FunSpec() {
   init {
      List(10000) { it }.forEach {
         test("test $it") {
            1 shouldBe 1
         }
      }
   }
}
Try that for me
s
you can see the full codebase here. here is the multiline string test, https://github.com/snowe2010/pretty-print/blob/master/src/test/kotlin/com/tylerthrailkill/helpers/prettyprint/MultilineStringTest.kt#L73 and here is the linebreak file https://github.com/snowe2010/pretty-print/blob/master/src/test/resources/LineBreakTest.txt, but note that that's all before the conversion to kotest.
s
If you try the test case I just pasted. Takes 5 seconds for me.
If you run that in gradle I think it will take a long time (no need to run it to completion, just see if it's lagging)
s
takes 10s in intellij, already over 30 seconds in gradle and that's only 1500 cases.
s
yeah its lagging for me in gradle
so it's almost certainly junit listener
let me fix and push a snapshot and we can try that
s
oh sweet!
woah.... I let that test case run to completion and it ran 99 extra tests....
s
that's odd
s
it's like it ran an extra test for every 100 tests...
s
better safe than sorry ?
s
hahaha
s
I've fixed it. Just writing a test.
s
awesome. let me know when to test on my end!
s
Ok give 4.3.0.621-SNAPSHOT a go
don't forget to add the snapshots maven repo
s
sorry, got super busy. trying now
awesome. 3.9s for 7284 tests. Thanks so much!
ah wait, I ran it using the intellij runner.
😂 1
darn still slow. much faster than before though... ran about 4k tests in 60 seconds. but is slowing down from there. 6700 tests in 2 minutes.
🧐 1
s
you're definitely on the snapshot everywhere
s
yeah let me post my dependencies. I can also push my branch if you want to try itout.
s
ok that would help yeah
I'm sure it will be something simple, and I'll be able to fix it quickly
s
correction! I had changed the version of the kotest-pitest plugin lol. after changing the correct version it builds in 37 seconds, which is a major improvement! It still doesn't run as fast as intellij standalone, but that is fine.
I was distracted when changing the version number and then subsequently responding to you. Sorry about that. 🤦🏽
s
the difference between intellij and junit will be due to the way the junit output writer works
but if it only took 2 seconds in spek, and it takes 37 seconds now, then that's not good enough
assuming the tests were identical
s
let me push my branch then and get you the spek version as well. I think the spek tests were partially borked on master.
and yeah I literally just replaced 
Spek
with 
DescribeSpek
and removed the spek setup function I had.
s
2 seconds does seem pretty fast for 7k tests
s
yeah it does actually... but it runs in 4 seconds on kotest in intellij so maybe not that farfetched.
s
true
I get 5 seconds if I run via gradle mind (on my dummy loop 10k loop test)
There could be another cause of slowness. What's your map to method doing.
well I can take a look when you push
s
just realized I had parallelism set in the kotest config. let me get rid of that real quick and see how fast it is.
actually I think it's the compilation step in intellij that's taking so long. I'll still push my branch for you. one se.
ok, I ran it with just this command and it only took 10s! so that's super good!
Copy code
❯ ./gradlew test --tests="com.tylerthrailkill.helpers.prettyprint.MultilineStringTest"

> Configure project :
Inferred project: pretty-print, version: 2.1.0-dev.15.uncommitted+kotest.21f0e7a

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See <https://docs.gradle.org/6.5.1/userguide/command_line_interface.html#sec:command_line_warnings>

BUILD SUCCESSFUL in 10s
6 actionable tasks: 2 executed, 4 up-to-date
oh that wasn't the greatest output, but yeah you get it. it did actually run all the tests.
s
ok perf
not much for me to look at then
Why did you migrate from spek out of interest ?
s
honestly, I tried kotest out and it immediately was a much better experience. Migration was the easiest migration I've ever seen. There was only a single dependency to add, whereas with spek I had to exclude certain dependencies. I also had to create special setup functions for spek that I don't need at all in kotest (no clue why. I needed memoization in a spek setup function in order to test properly). The spek intellij plugin fails all the time, and is pretty slow. The kotest plugin is incredibly fast and worked perfectly first time. That alone was enough for me. One thing I didn't like (initially) was that spek allows me to have duplicate test names, kotest does not. That 7k line file from before had quite a lot of duplicates, that I didn't know about until switching to kotest hahahaha. That was a one time fix though, but it did take a while, because kotest fails on the first duplicate, instead of finding all duplicates. Also, kotest doesn't output the duplicate name. I had to debug into kotest and put a watch to see what the name was just to delete the right line from the LineBreakTest.txt file. Finally, you have a PIT plugin. Spek is completely incompatible with PIT and I think mutation tests are super powerful. That alone would have been enough for me to switch, with none of the other benefits. Even if the experience had been worse I still would have switched.
s
Right, that's really cool. I think kotest is the best test framework but of course I'm heavily biased.
As for Also, kotest doesn't output the duplicate name, good shout, I'll check that now and fix it
s
awesome! Hopefully that helps someone else further down the road. And hopefully me if I ever update that linebreak file 😂
also, I actually love the FreeSpec style. It looks so good in nested tests with forAll/rows.
s
I never liked that one so much 🙂
s
I actually didn't like it when I first saw it as well, but then decided to try it out and it's crazy readable in certain contexts. I could definitely see how it wouldn't be readable or nice in other contexts.
s
I was a scala developer for 9 years.
s
Ah. I can definitely see how it would not be great in that context. My current team is me and one other dev. So I think we will be able to maintain it pretty decently. Also we were hired to literally trash the monolith we're working on so all the FreeSpec stuff will be gone eventually lol. Maybe I'll figure out if I hate it before then and decide to use a different form. What's your favorite?
s
I wasn't trying to convince you to not use it 🙂 I think free spec is very popular, because it's good when you want to generate tests like you are.
FunSpec is my favourite because I'm old and like to do things without nesting
s
ah. haha. I think nesting is the most powerful test construct. I think it makes stuff way more readable and refactorable. I know a lot of people don't like it though.
s
If I do end up with some nesting for whatever reason, I like describe spec
because it copies the javascript style
s
yeah that was the one I started off using. I also code a lot in ruby so used to that as well.
s
I think it started off with rspec and then the js frameworks borrowed it
s
yeah.
anyway, thanks for the hard work! I love your library!
❤️ 1
l
FunSpec is my favourite because I'm old and like to do things without nesting
Particularly, I use ShouldSpec for almost every scenario
6 Views
Open in Slack
PreviousNext
Powered by