Is there a way I can configure Test Balloon as the...

# kotlin-toolchain

[gbaldeck]

Is there a way I can configure Test Balloon as the test runner rather than Kotlin Test? I currently have the required library and compiler plugin in my configuration. But the gutter icons don't work to run the tests with the Test Balloon IntelliJ plugin installed.

[joffrey]

I'm not super familiar with TestBalloon myself, but I know the author @Oliver.O managed to get things working with the Kotlin Toolchain. Oliver, maybe you can help me out, here? Is there any IDE support so far for projects using the Kotlin Toolchain?

[joffrey]

@gbaldeck could you please confirm whether your tests run correctly with the CLI?

[Oliver.O]

Thanks Joffrey for pinging me! The IDE plugin story needs to be investigated, as there is some extra work required to create Kotlin Toolchain run configurations instead of Gradle run configurations. So currently, the plugin does not support run gutter icons for Kotlin Toolchain.

[gbaldeck]

@joffrey kotlin test -p jvm is not discovering my TestBalloon tests. I am getting the below error
ERROR: JVM tests failed for module 'shared-frontend' with exit code 2 (no tests were discovered) (see errors above)

[gbaldeck]

Here is my template

[gbaldeck]

Here is my test file

[gbaldeck]

@Oliver.O Were you able to configure TestBalloon to run with the toolchain CLI?

[Oliver.O]

It's been a while when I last checked, but I can try again.

[joffrey]

Yeah the original success story was there:
https://jetbrains.slack.com/archives/C062WG3A7T8/p1775820943237589?thread_ts=1775645892.567979&cid=C062WG3A7T8

[joffrey]

@Oliver.O is the TestBalloon JUnit engine part of testBalloon-framework-core or is it a separate dependency?

[Oliver.O]

It's part of testBalloon-framework-core.

[joffrey]

Then I don't get why it's not picked up. @gbaldeck, is it possible for you to share a reproducer project?

[Oliver.O]

I re-checked again. TestBalloon does run tests with Kotlin Toolchain on the JVM, but it needs the following environment variable:

test-settings:
  jvm:
    test:
      extraEnvironment:
        - TESTBALLOON_INCLUDE_PATTERNS: '*'

Without that setting, TestBalloon presumably receives an unexpected "test discovery request" via JUnit Platform, leading it to believe that tests were requested for a different "test engine", and not try to run them on its own.

My entire module was looking like this:

product:
  type: lib
  platforms:
    - jvm
#    - linuxX64
#    - android

settings@android:
  junit: junit-4

test-dependencies:
  - de.infix.testBalloon:testBalloon-framework-core:1.1.0-RC

test-settings:
  jvm:
    test:
      extraEnvironment:
        - TESTBALLOON_INCLUDE_PATTERNS: '*'
  kotlin:
    version: 2.4.10
    compilerPlugins:
      - id: de.infix.testBalloon
        dependency: de.infix.testBalloon:testBalloon-compiler-plugin:1.1.0-RC

[joffrey]

Ah, I forgot you skip the discovery params from JUnit. I wonder how we can make that work nicer with our CLI options for the test command. We have --include-test, --include-classes, --include-tag there, but they are all passed to JUnit

[joffrey]

How is the build tool expected to pass filters to testballoon? Via this env var? What's the format?

[Oliver.O]

It's the old story: With Gradle, none of these options worked on the JVM via JUnit, as TestBalloon supports a multi-level hierarchiy for tests which Gradle's test filters still do not support. (Gradle got some hierarchy support lately, but only for reporting.)

So the safest way to support TestBalloon's test selection (a.k.a. filtering) is via environment variables TESTBALLOON_INCLUDE_PATTERNS and TESTBALLOON_EXCLUDE_PATTERNS. if TESTBALLOON_INCLUDE_PATTERNS is present, TestBalloon will know that it is in charge, so setting it to a wildcard * makes tests work. For Gradle, the TestBalloon Gradle plugin takes the argument of --tests and passes its value unchanged to TESTBALLOON_INCLUDE_PATTERNS. The format is documented here: infix-de.github.io/testBalloon/dev/getting-started/integration#…

Of course, we could also agree on a different protocol, like the `UriSelector`.

[Oliver.O]

An updated working example project is here: github.com/OliverO2/kotlin-toolchain-arena

[gbaldeck]

I tried the above and looked at the sample project, but still no luck. I am getting the same error that my tests cant be found. As far as I can tell I have everything setup correctly. I will share my project.

[gbaldeck]

Here is the project. I am attempting to run kotlin test -p jvm . If you run the same you should see ERROR: JVM tests failed for module 'shared-frontend' with exit code 2 (no tests were discovered) (see errors above) .

The shared-frontend module is where the TestBalloon tests are. Its module.yaml file inherits cmp.module-template.yaml . Each module file inherits from a template.

My base.module-template.yaml is where TestBalloon is setup. Then ai.module-template.yaml inherits from the base template. Then cmp.module-template.yaml inherits from the ai template.

[gbaldeck]

I also tried by setting up a run configuration and pointing it to my shared-frontend module. That also didn't work when running the test command, including the TESTBALLOON_INCLUDE_PATTERNS environment variable, and setting the platform to JVM.

[Oliver.O]

Unfortunately, that project is a bit too heavy to analyze. I can't even determine if it's safe to run, as it might download and execute untrusted code (AI agents).

I'd suggest to eliminate complexity first, e.g. by configuring the shared-frontend without templates, then add a single template, ..., and then see where it breaks.

One thing that might help is turning on the TestBalloon compiler plugin's debug logging option by setting debugLevel: BASIC. The full set of settings would look like this:

test-settings:
  jvm:
    test:
      extraEnvironment:
        - TESTBALLOON_INCLUDE_PATTERNS: '*'
  kotlin:
    version: 2.4.10
    compilerPlugins:
      - id: de.infix.testBalloon
        dependency: de.infix.testBalloon:testBalloon-compiler-plugin:1.1.0-RC
        options:
          debugLevel: BASIC

You should then expect to see messages like this on the console (the ERROR tag can be ignored):

ERROR warning: plugin de.infix.testBalloon: [DEBUG] using compiler adapter 2.4.0 for Kotlin compiler 2.4.10
ERROR warning: plugin de.infix.testBalloon: [DEBUG] version 1.1.0-RC is processing module <usingTestBalloon_test>.
ERROR .../kotlin-toolchain-arena/usingTestBalloon/test/CommonTestBalloon.kt:1:1: warning: plugin de.infix.testBalloon: [DEBUG] Analyzing source file.
ERROR de/infix/testBalloon/framework/shared/internal/entryPoint/__GENERATED__CALLABLES__.kt: warning: plugin de.infix.testBalloon: [DEBUG] Analyzing source file.
✓ Compilation successful for 'usingTestBalloon' [linuxX64 tests]
WARNING: Plugin de.infix.testBalloon: [DEBUG] version 1.1.0-RC is processing module <usingTestBalloon_test>.
WARNING: Plugin de.infix.testBalloon: [DEBUG] using compiler adapter 2.4.0 for Kotlin compiler 2.4.10
WARNING:  Plugin de.infix.testBalloon: [DEBUG] Integrating with JUnit

[gbaldeck]

I realize I am pretty early in the kotlin-toolchain adoption but projects like the one I have setup should be expected to be common place. It has a server, frontend, and shared module. That's it.

Then the front end is split into the respective client apps (like the toolchain prescribes) and a shared module between them for CMP.

The Koog AI libraries go into the shared module between the frontend and server because for the wasm frontend app the AI agent will need to be run on the server where is it can be run locally for the frontend desktop, android, and ios apps.

Thanks for the suggestions I may give them a try, but have switched to Kotlin Test for now in the interest of moving along with the project.

[Oliver.O]

Stripped down your project and found the cause:

libs.versions.toml declared a Gradle plugin

test-balloon-plugin = { module = "de.infix.testBalloon:de.infix.testBalloon.gradle.plugin", version.ref = "testBalloon"}

when it should have declared a compiler plugin

test-balloon-plugin = { module = "de.infix.testBalloon:testBalloon-compiler-plugin", version.ref = "testBalloon"}

[Oliver.O]

@joffrey There should be a better diagnostic for this case. Could the compiler better flag a non-plugin dependency or is Kotlin Toolchain in a better position to do so?

[joffrey]

That is tricky. If anything I think it would have to be in the compiler. Technically, compiler plugins can also bring a transitive graph of dependencies, so not all jars have to contain a plugin. As a build tool, we just resolve a dependency graph based on the given Maven coordinates of the compiler plugin, and pass the resulting JARs to the compiler, but it's opaque for us. We can't know which are plugins and which aren't. That's also why we need to be given the ID of the compiler plugin in addition to the JAR - we don't know what's inside the jar.

[joffrey]

But even for the compiler, at the end of the day it receives a bunch of JAR, and technically they don't all have to contain a compiler plugin. So it can't really be an error. At least, that is true for how the compiler works right now (with one -Xplugin for each JAR, so potentially several -Xplugin for a single compiler plugin). Maybe in the future we'll get a new compiler argument that takes the whole little classpath for each compiler plugin independently, and then it could warn or fail if there are 0 plugins in one of these little classpaths.

[Oliver.O]

Sure, I haven't put a lot of thoughts into how it could be done. So my wild guess was that if something is declared a compiler plugin dependency, it would have to contribute a META-INF.services resource of org.jetbrains.kotlin.compiler.plugin.CompilerPluginRegistrar at some point.

[joffrey]

Yeah so indeed we could decide not to consider these JARs opaque, and actually make use of this knowledge to unpack the JARs and look for service resources inside. I don't know the impact it would have on performance, but it might be reasonable. We would have to inspect potentially all JARs in the tree because nothing forbids to bring the plugin transitively, but since 99.99% of the time it will be the root node, we can probably consider that the effect would just be that of reading a single entry from a single JAR.

[joffrey]

The only other cost would be that we couple the build tool to how compiler plugins are structured in side, but I don't think that's a big deal in the grand scheme of things 🙂

[Oliver.O]

For now, it’s just an idea, and maybe it can wait for a while, because downloadable Toolchain plugins would make this less of an issue. 🙂

[joffrey]

Yes, sure, but worth exploring. This is exactly the kind of friction we want to eliminate if we can help it. I think your idea makes sense. I opened this issue for tracking:
KTC-5836 Fail early if a compiler plugin dependency is not a compiler plugin

[Oliver.O]

This is exactly the kind of friction we want to eliminate if we can help it.

I love that stance!