Also I noticed that since upgrading to Spek 2.0.3 the amount of test results drastically reduced in ...

# spek

[svenjacobs]

Also I noticed that since upgrading to Spek 2.0.3 the amount of test results drastically reduced in Jenkins from 1000 to a few hundreds. After upgrading to 2.0.4 even less tests are reported. Did something change how Spek reports JUnit test results?

[raniejade]

Nothing changed on the reporting side. But then again there is a critical bug introduced in 2.0.3, exceptions were not propagated properly (failed tests are being reported as passed). This bug was fixed in 2.0.4. Are you able to share a reproducer?

[svenjacobs]

I compared the output of ./gradlew test between Spek 2.0.2 and 2.0.3 and don't see a difference. However Jenkins has "lost" many test results since the Spek upgrade. Not sure why...

[raniejade]

Weird, no upgrade was done to JUnit 5 dependency (which is responsible for reporting).

[tapchicoma]

noticed this as well after upgarding to 2.0.4 version:
now report doesn't contain any text from context(..) {} and describe {} methods, only from it(..) {}

[svenjacobs]

Still trying to figure out why suddenly many tests are not discovered anymore. First I thought this is only an issue on Jenkins but today I realized that it happens on my local machine, too. This wasn't apparent immediatly because the test work when running inside the IDE. But ./gradlew test does only discover a portion of the tests. Also I'm not really sure what the pattern is here. For instance I have the modules core/common and core/domain. Both modules have the exact same configuration. However tests are only discovered for core/common?! I'm really scratching my head here 🤯

[raniejade]

Hey! Would it be possible to share the project?

[svenjacobs]

Unfortunately not. It's a very complex and commercial project. I tried to reproduce this in an example project but was not able to reproduce the issue yet.

[raniejade]

Have you tried running the test for a specific module? Something like: ./gradlew coredomaintest..

[svenjacobs]

Yes, tests are not discovered

[raniejade]

Have you tried running it with —info or —debug?

[svenjacobs]

./gradlew :core:common:test works but ./gradlew :core:domain:test does not. Does not work means that the build succeeds. However tests are not run. I forced an AssertionError in one of the tests under domain but the error is not reported

[raniejade]

Discovery failures are silently consumed by junit 5

[raniejade]

Best bet is one of the test classes is causing discovery failures. Junit 5 would just print an error but never propagate that failure to gradle

[svenjacobs]

Ooops, this seems to be the case. I just ran with --info and there is a discovery error produced by some library I use in domain.

[svenjacobs]

Really odd that this is not reported as an error

[svenjacobs]

but says build runs fine

[raniejade]

Yeah, not sure if it’s a problem with junit 5 or how gradle implemented support for it. Could be both 🤷‍♂️

[raniejade]

https://github.com/junit-team/junit5/issues/242 is what we need.

[svenjacobs]

Oh, that's and old ticket 😮 Just upvoted it