I'm trying to use <https://docs.junit.org/6.0.3/ru...
# kotest
s
I'm trying to use https://docs.junit.org/6.0.3/running-tests/console-launcher.html#options-executing-tests with the Kotest engine. But while Kotest as the engine is discovered, and also a "container" (whatever that is), no tests are discovered or executed when I use options
execute --scan-classpath --include-engine=kotest
. Any idea why that is? (More output in thread.)
Copy code
❯ ./gradlew -q :cli-test-launcher:run --args="execute --scan-classpath --include-engine=kotest"

Thanks for using JUnit! Support its development at <https://junit.org/sponsoring>

19:03:46,003 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found logback-core version 1.5.32
19:03:46,005 |-INFO in ch.qos.logback.classic.util.ContextInitializer@15383681 - No custom configurators were discovered as a service.
19:03:46,005 |-INFO in ch.qos.logback.classic.util.ContextInitializer@15383681 - Trying to configure with ch.qos.logback.classic.util.DefaultJoranConfigurator
19:03:46,006 |-INFO in ch.qos.logback.classic.util.ContextInitializer@15383681 - Constructed configurator of type class ch.qos.logback.classic.util.DefaultJoranConfigurator
19:03:46,008 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
19:03:46,010 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/home/sebastian/Development/GitHub/oss-review-toolkit/ort/cli/build/resources/main/logback.xml]
19:03:46,010 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@743e66f7 - Resource [logback.xml] occurs multiple times on the classpath.
19:03:46,010 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@743e66f7 - Resource [logback.xml] occurs at [file:/home/sebastian/Development/GitHub/oss-review-toolkit/ort/cli-helper/build/resources/main/logback.xml]
19:03:46,010 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@743e66f7 - Resource [logback.xml] occurs at [file:/home/sebastian/Development/GitHub/oss-review-toolkit/ort/cli/build/resources/main/logback.xml]
19:03:46,010 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@743e66f7 - Resource [logback.xml] occurs at [jar:file:/home/sebastian/Development/GitHub/oss-review-toolkit/ort/utils/test/build/libs/test-utils-84.0.0-004.sha.2ad27e4.jar!/logback.xml]
19:03:46,076 |-INFO in ch.qos.logback.classic.model.processor.ConfigurationModelHandlerFull - Scan attribute not set or set to unrecognized value.
19:03:46,081 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [STDOUT]
19:03:46,081 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
19:03:46,088 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
19:03:46,123 |-INFO in ch.qos.logback.core.ConsoleAppender[STDOUT] - NOTE: Writing to the console can be slow. Try to avoid logging to the 
19:03:46,123 |-INFO in ch.qos.logback.core.ConsoleAppender[STDOUT] - console in production environments, especially in high volume systems.
19:03:46,123 |-INFO in ch.qos.logback.core.ConsoleAppender[STDOUT] - See also <https://logback.qos.ch/codes.html#slowConsole>
19:03:46,124 |-INFO in ch.qos.logback.classic.model.processor.RootLoggerModelHandler - Setting level of ROOT logger to WARN
19:03:46,124 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[ROOT]
19:03:46,125 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.http.headers] to ERROR
19:03:46,125 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.http.wire] to ERROR
19:03:46,125 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.eclipse.jgit.internal.storage.file.FileSnapshot] to ERROR
19:03:46,125 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.analyzer.managers.Yarn2] to INFO
19:03:46,125 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.clients.fossid.LoggingInterceptor] to INFO
19:03:46,125 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.clients.fossid.FossIdRestService] to INFO
19:03:46,125 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.plugins.reporters.fossid.FossIdReporter] to INFO
19:03:46,125 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.plugins.reporters.fossid.FossIdSnippetReporter] to INFO
19:03:46,125 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.plugins.scanners.fossid.FossId] to INFO
19:03:46,125 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.plugins.scanners.fossid.FossIdConfig] to INFO
19:03:46,125 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.plugins.scanners.fossid.FossIdUrlProvider] to INFO
19:03:46,125 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.plugins.scanners.fossid.events.CloneRepositoryHandler] to INFO
19:03:46,125 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.plugins.scanners.fossid.events.UploadArchiveHandler] to INFO
19:03:46,126 |-INFO in ch.qos.logback.core.model.processor.DefaultProcessor@109a2025 - End of configuration.
19:03:46,126 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@2241f05b - Registering current configuration as safe fallback point
19:03:46,129 |-INFO in ch.qos.logback.classic.util.ContextInitializer@15383681 - ch.qos.logback.classic.util.DefaultJoranConfigurator.configure() call lasted 120 milliseconds. ExecutionStatus=DO_NOT_INVOKE_NEXT_IF_ANY

╷
└─ Kotest ✔

Test run finished after 94 ms
[         1 containers found      ]
[         0 containers skipped    ]
[         1 containers started    ]
[         0 containers aborted    ]
[         1 containers successful ]
[         0 containers failed     ]
[         0 tests found           ]
[         0 tests skipped         ]
[         0 tests started         ]
[         0 tests aborted         ]
[         0 tests successful      ]
[         0 tests failed          ]
Looks like the problem is this code in Junit. It only discovers classes from the
main
source set, not from my custom
funTest
source set.
s
I think what's happening is the console launcher uses a selector kotest doesn't support. I'll raise a pr tonight to add this functionality
s
Hmm, I'm not yet sure whether something Kotest-specific is going on or not. It seems that already before JUnit calls the Kotest engine, the selectors do not include the
funTest
source set.
Ah, sorry, I misread your comment.
s
I bet they send a package selector
Well I'm guessing
s
It's `ClasspathRootSelectors`:
s
There you go
And we're supposed to look it up
So I'll add a class path scan
We do that anyway for the kotest task, but for junit gradle normally provides it as class selectors
s
I thought the whole point is that JUnit does the classpath scanning for you, so Kotest does not have to. Wasn't classpath scanning just removed in Kotest 6?
s
Yes that's the case for gradle, my guess is the console launcher doesn't do that? Could be wrong of course
It's gradle that scans not junit when you run gradlew test
💡 1
s
I might be wrong as well, but as the JUnit console has a
--scan-classpath
option, I though it would be doing that for you.
s
Ah ok
Well I'm not at my computer so I'm probably taking out my arse lol
I will investigate tonight though
s
Thanks!
s
Is this on a branch because on main you're using the Kotest launcher directly it seems?
s
Yes, let me push it. I was looking for an alternative solution as using the Kotest launcher directly requires me to provide the specs arg.
If there was a way to simply run all tests with the Kotest launcher without the need to explicitly specify them, that would also work for me.
Anyway, I just pushed the junit-launcher branch.
s
ok great thanks
👍🏻 1
s
I'm still puzzled why all
ClasspathRootSelectors
that JUnit determines point to the
main
source set only. Even if Kotest would do the actual scanning, start from those roots it'd never discover my
funTests
. So I'm rather looking into the direction of a JUnit console misconfiguration.
s
after playing around I agree with you - we already implemented classpathroot selectors
s
Do you think it would be feasible to implement an option to the Kotest launcher directly to run all test on the classpath? Then I would not have to use the JUnit launcher at all.
s
yes, that's easy enough to do
s
Great, just some CLI option to the main function would be awesome!
@sam how would you propose to implement this exactly? Should the
spec
arg for the launcher maybe accept a special "ALL" value that makes the launcher itself discover all tests on the classpath? Or should a completely separate argument be introdiced for that, which is mutualls exclusive with
spec
?
s
I think we can definitely have the launcher use classpath scanning if no specs are provided - but I wonder if the same issue will happen, that the classpath doesn't contain /test ? Because when I dump the entire classpath roots in your project, only main is included. So I wonder if it's just a gradle config thing that needs to be tweaked?
s
Weird. This line should take care of funTests being included. And if I do e.g.
./gradlew :cli-test-launcher:installDist
, I do see
*-fun-test.jar
files in
lib
of the distribution. So I think it should be fine.
s
sorry that there's been no progress on this - will add it to the weekends tasks
s
No worries, I'm grateful you're considering to implement this! It's open source after all, so also a bit of scratch-your-own-itch if you want to get things implemented 🙂