https://kotlinlang.org logo
Title
s

snowe

09/04/2020, 12:30 AM
i have a generative test case that generates over 7k tests. When run by itself in intellij (just the test class), the test completes in 4 seconds. If instead, I select the directory and run all tests in the whole project, the tests take over 22 minutes to complete. Each consecutive test takes longer than the one before to complete. No clue what is going on here. anyone ever seen this?
s

sam

09/04/2020, 1:32 AM
What do you mean by generative test case
s

snowe

09/04/2020, 1:34 AM
here's my exact test case:
context("break should occur between ") {
            javaClass.getResource("/LineBreakTest.txt")
                .readText()
                .lines()
                .forEach nextTest@{ testLine ->
                if (testLine.startsWith('#') or testLine.isBlank()) {
                    return@nextTest
                }
                val parts = mapUnicodeTestLineToParts(testLine)
                val padding = "                                       "
                val testName = testLine.split('#')[1]
                if (testsToSkipCurrently.contains(testName)) {
                    return@nextTest
                }
                it(testName) {
                    <http://logger.info|logger.info> { testName }
                    logger.debug { parts }
                    prettyPrint(
                        wrappedLineWidth = 1,
                        obj = LongString(parts.flatten().joinToString(""))
                    ) mapTo """
                        LongString(
                          longString = ""${'"'}
                                       ${parts.joinToString("\n$padding") { it.joinToString("") }}
                                       ""${'"'}
                        )
                        """
                }
            }
        }
s

sam

09/04/2020, 1:39 AM
return@nextTest
where's nextText
Is this a simplified example because you're not actually doing any assertions in it
s

snowe

09/04/2020, 1:43 AM
nextTest is the name of the
forEach
at the top. And the assertion is in mapTo.
s

sam

09/04/2020, 1:44 AM
ok I see now yeah
and linebreaks.txt has about 7k lines
s

snowe

09/04/2020, 1:44 AM
yep
s

sam

09/04/2020, 1:45 AM
and even if they all pass it takes forever
s

snowe

09/04/2020, 1:45 AM
yeah.
s

sam

09/04/2020, 1:45 AM
sounds like an o(n) issue in the junit listener
class FooTest : FunSpec() {
   init {
      List(10000) { it }.forEach {
         test("test $it") {
            1 shouldBe 1
         }
      }
   }
}
Try that for me
s

snowe

09/04/2020, 1:46 AM
you can see the full codebase here. here is the multiline string test, https://github.com/snowe2010/pretty-print/blob/master/src/test/kotlin/com/tylerthrailkill/helpers/prettyprint/MultilineStringTest.kt#L73 and here is the linebreak file https://github.com/snowe2010/pretty-print/blob/master/src/test/resources/LineBreakTest.txt, but note that that's all before the conversion to kotest.
s

sam

09/04/2020, 1:46 AM
If you try the test case I just pasted. Takes 5 seconds for me.
If you run that in gradle I think it will take a long time (no need to run it to completion, just see if it's lagging)
s

snowe

09/04/2020, 1:48 AM
takes 10s in intellij, already over 30 seconds in gradle and that's only 1500 cases.
s

sam

09/04/2020, 1:48 AM
yeah its lagging for me in gradle
so it's almost certainly junit listener
let me fix and push a snapshot and we can try that
s

snowe

09/04/2020, 1:48 AM
oh sweet!
woah.... I let that test case run to completion and it ran 99 extra tests....
s

sam

09/04/2020, 1:56 AM
that's odd
s

snowe

09/04/2020, 1:57 AM
it's like it ran an extra test for every 100 tests...
s

sam

09/04/2020, 1:58 AM
better safe than sorry ?
s

snowe

09/04/2020, 1:58 AM
hahaha
s

sam

09/04/2020, 2:05 AM
I've fixed it. Just writing a test.
s

snowe

09/04/2020, 2:19 AM
awesome. let me know when to test on my end!
Ok give 4.3.0.621-SNAPSHOT a go
don't forget to add the snapshots maven repo
s

snowe

09/04/2020, 3:53 AM
sorry, got super busy. trying now
awesome. 3.9s for 7284 tests. Thanks so much!
ah wait, I ran it using the intellij runner.
😂 1
darn still slow. much faster than before though... ran about 4k tests in 60 seconds. but is slowing down from there. 6700 tests in 2 minutes.
🧐 1
s

sam

09/04/2020, 4:01 AM
you're definitely on the snapshot everywhere
s

snowe

09/04/2020, 4:03 AM
yeah let me post my dependencies. I can also push my branch if you want to try itout.
s

sam

09/04/2020, 4:03 AM
ok that would help yeah
I'm sure it will be something simple, and I'll be able to fix it quickly
s

snowe

09/04/2020, 4:14 AM
correction! I had changed the version of the kotest-pitest plugin lol. after changing the correct version it builds in 37 seconds, which is a major improvement! It still doesn't run as fast as intellij standalone, but that is fine.
I was distracted when changing the version number and then subsequently responding to you. Sorry about that. 🤦🏽
s

sam

09/04/2020, 4:15 AM
the difference between intellij and junit will be due to the way the junit output writer works
but if it only took 2 seconds in spek, and it takes 37 seconds now, then that's not good enough
assuming the tests were identical
s

snowe

09/04/2020, 4:16 AM
let me push my branch then and get you the spek version as well. I think the spek tests were partially borked on master.
and yeah I literally just replaced
Spek
with
DescribeSpek
and removed the spek setup function I had.
s

sam

09/04/2020, 4:17 AM
2 seconds does seem pretty fast for 7k tests
s

snowe

09/04/2020, 4:17 AM
yeah it does actually... but it runs in 4 seconds on kotest in intellij so maybe not that farfetched.
s

sam

09/04/2020, 4:17 AM
true
I get 5 seconds if I run via gradle mind (on my dummy loop 10k loop test)
There could be another cause of slowness. What's your map to method doing.
well I can take a look when you push
s

snowe

09/04/2020, 4:19 AM
just realized I had parallelism set in the kotest config. let me get rid of that real quick and see how fast it is.
actually I think it's the compilation step in intellij that's taking so long. I'll still push my branch for you. one se.
ok, I ran it with just this command and it only took 10s! so that's super good!
❯ ./gradlew test --tests="com.tylerthrailkill.helpers.prettyprint.MultilineStringTest"

> Configure project :
Inferred project: pretty-print, version: 2.1.0-dev.15.uncommitted+kotest.21f0e7a

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See <https://docs.gradle.org/6.5.1/userguide/command_line_interface.html#sec:command_line_warnings>

BUILD SUCCESSFUL in 10s
6 actionable tasks: 2 executed, 4 up-to-date
oh that wasn't the greatest output, but yeah you get it. it did actually run all the tests.
s

sam

09/04/2020, 4:25 AM
ok perf
not much for me to look at then
Why did you migrate from spek out of interest ?
s

snowe

09/04/2020, 4:32 AM
honestly, I tried kotest out and it immediately was a much better experience. Migration was the easiest migration I've ever seen. There was only a single dependency to add, whereas with spek I had to exclude certain dependencies. I also had to create special setup functions for spek that I don't need at all in kotest (no clue why. I needed memoization in a spek setup function in order to test properly). The spek intellij plugin fails all the time, and is pretty slow. The kotest plugin is incredibly fast and worked perfectly first time. That alone was enough for me. One thing I didn't like (initially) was that spek allows me to have duplicate test names, kotest does not. That 7k line file from before had quite a lot of duplicates, that I didn't know about until switching to kotest hahahaha. That was a one time fix though, but it did take a while, because kotest fails on the first duplicate, instead of finding all duplicates. Also, kotest doesn't output the duplicate name. I had to debug into kotest and put a watch to see what the name was just to delete the right line from the LineBreakTest.txt file. Finally, you have a PIT plugin. Spek is completely incompatible with PIT and I think mutation tests are super powerful. That alone would have been enough for me to switch, with none of the other benefits. Even if the experience had been worse I still would have switched.
s

sam

09/04/2020, 4:33 AM
Right, that's really cool. I think kotest is the best test framework but of course I'm heavily biased.
As for Also, kotest doesn't output the duplicate name, good shout, I'll check that now and fix it
s

snowe

09/04/2020, 4:34 AM
awesome! Hopefully that helps someone else further down the road. And hopefully me if I ever update that linebreak file 😂
also, I actually love the FreeSpec style. It looks so good in nested tests with forAll/rows.
s

sam

09/04/2020, 4:38 AM
I never liked that one so much 🙂
s

snowe

09/04/2020, 4:39 AM
I actually didn't like it when I first saw it as well, but then decided to try it out and it's crazy readable in certain contexts. I could definitely see how it wouldn't be readable or nice in other contexts.
s

sam

09/04/2020, 4:39 AM
I was a scala developer for 9 years.
s

snowe

09/04/2020, 4:42 AM
Ah. I can definitely see how it would not be great in that context. My current team is me and one other dev. So I think we will be able to maintain it pretty decently. Also we were hired to literally trash the monolith we're working on so all the FreeSpec stuff will be gone eventually lol. Maybe I'll figure out if I hate it before then and decide to use a different form. What's your favorite?
s

sam

09/04/2020, 4:43 AM
I wasn't trying to convince you to not use it 🙂 I think free spec is very popular, because it's good when you want to generate tests like you are.
FunSpec is my favourite because I'm old and like to do things without nesting
s

snowe

09/04/2020, 4:45 AM
ah. haha. I think nesting is the most powerful test construct. I think it makes stuff way more readable and refactorable. I know a lot of people don't like it though.
s

sam

09/04/2020, 4:45 AM
If I do end up with some nesting for whatever reason, I like describe spec
because it copies the javascript style
s

snowe

09/04/2020, 4:47 AM
yeah that was the one I started off using. I also code a lot in ruby so used to that as well.
s

sam

09/04/2020, 4:47 AM
I think it started off with rspec and then the js frameworks borrowed it
s

snowe

09/04/2020, 4:48 AM
yeah.
anyway, thanks for the hard work! I love your library!
❤️ 1
l

LeoColman

09/05/2020, 9:08 PM
FunSpec is my favourite because I'm old and like to do things without nesting
Particularly, I use ShouldSpec for almost every scenario