https://kotlinlang.org logo
Join Slack
Powered by
I'm investigating how we can translate some VS Cod...
# testballoon
e
I'm investigating how we can translate some VS Code TS integration tests to Kotlin/JS. The entry point the VS Code executor mandates is:
Copy code
export async function run(): Promise<void> {
  const mocha = new Mocha({ ... });
  // Omitted
  return new Promise<void>((resolve, reject) => {
    // Run the mocha test suite
    mocha.run((failures) => {
      if (failures === 0) {
        resolve();
      } else {
        reject(new Error(`${failures} tests failed.`));
      }
    });
  });
}
Where a promise is rejected in case of test case failures. Does TestBalloon support registering/discovering tests through a custom function? I did investigate this a long time ago, and couldn't figure out anything with 
kotlin.test
.
o
TestBalloon supports registering tests and test suites via custom functions. More details are explained here. Aspects to consider: • All tests and suites must be registered before execution starts. See Green code and blue code. • TestBalloon, always trying to be a good citizen in the Kotlin ecosystem, uses the Kotlin JS test infra, which means that tests run with Mocha on JS.
e
registering tests and test suites via custom functions
That happens under a test suite scope tho, right? Inside my 
export async function run(): Promise<void>
function I will have to register TestBalloon test suites instead of test cases.
I think what I'm really missing in the picture is how to get both worlds working. 1. being able to register and execute test suites in the normal Kotlin workflow 2. being able to register and execute the same test suites through that custom entry point
I was thinking about something like:
Copy code
// TestSuiteOne.kt
val TestSuiteOne by testSuite { ... }

// TestSuiteTwo.kt
val TestSuiteTwo by testSuite { ... }

// Run.kt
@JsExport
fun run(): Promise<Unit> {
  register(TestSuiteOne)
  register(TestSuiteTwo)
  // Somehow execute those suites
}
That means I'd be able to use both gutter icons, and VS Code run configurations.
o
That happens under a test suite scope tho, right?
Yes.
Copy code
// Somehow execute those suites
That would mean replacing the test framework, right? Discovery of top-level suites and creating an entry point (
main
-like function) is the job of TestBalloon's compiler plugin. Registration and execution is the primary job of TestBalloon's core library. Run gutter icons are the territory of TestBalloon's IDE plugin, which uses the IDE's and the Kotlin compiler's code analysis. And then there's integration (the rest of the owl).
Assume for a moment that we'd just have a 
run
entry point for VS Code in addition to the regular Kotlin/JS entry point. • How would VS Code handle test reporting (Kotlin/JS uses the TeamCity protocol for that)? • How would VS Code detect test suites and tests to provide run gutter icons?
e
Regarding your last comment about "how" in VS Code: there is nothing that fancy. VS Code displays the standard/error output from Mocha, and errors out when the returned promise is rejected.
I guess the test discovery phase should stay the same, it is just the "execution" phase that would have to change.
o
The primaty objective is to execute tests from VS Code instead of IntelliJ?
e
That's correct. But not through a special "test view". You can imagine a VS Code run config as simply calling a CLI command. The command then invokes that 
run
function, which executes the test suites.
o
And you wouldn't have run gutters in VS Code?
e
Nope. At least not standard. There are third party extensions that add that kind of stuff, but it's not built in.
It you want to see how extensions tests are run with custom runners, see https://code.visualstudio.com/api/working-with-extensions/testing-extension#the-test-runner-script
o
As far as I can tell, that would require replacing the low-level Kotlin/JS test infra, which both, kotlin.test and TestBalloon, use. The test infra uses Mocha, but doesn't produce Mocha output. It produces TeamCity protocol messages instead which are then picked up by the Kotlin Gradle plugin and converted to Gradle test events.
e
I believe that's workaroundable. I can write a wrapper over TC to deserialize messages and output them to stdout.
Messages are just base64 encoded IIRC.
But long term I believe the tight integration with IJ should disappear. Users want to use Kotlin with other IDEs too.
o
With TestBalloon, you could just add a 
TestConfig.executionReport
, so no need to deal with TeamCity conversion. But proper IDE integration is non-trivial and I wonder who‘d create a VS Code extension. What we covered here was just the surface of the JS part.
thank you color 1
e
You do not need a VSC extension, or a Zed extension, and so on. My own team uses a variety of IDEs and editors, and outside of IJ no one wants to deal with plugins. They prefer CLI interactions or doing their own scripting. So TestBalloon should just be flexible enough to be invoked outside of the IDE, once Kotlin has compiled the test code.
o
If you invoke it via Gradle or (soon) Amper, why not?
e
Invoking via Gradle/Amper is fine. But you're not considering the case where compiled code (let's think about JS again) is invoked by some other entity. In the case of VSC integration tests, the test code is invoked by the VSC test harness (development extension host), and Gradle is only used to compile.
It's like calling 
node tests.js
o
I guess it's really not TestBalloon: It's the core Kotlin distribution which currently is tied to Gradle, with Amper as an upcoming alternative, and TestBalloon just integrates into what exists. Suppose you're adventurous: If you manage to extract the JS test bundle from a Gradle build, you could just call a
Copy code
suspend fun main(arguments: Array<String>)
because that's what the TestBalloon compiler plugin generates for JS. With it, TestBalloon would run all its tests. If there is a Mocha-style test framework present (
describe
and 
it
functions), it would use that (not recommended for standalone execution). Otherwise, it would run them directly (better).
e
It's a mix of both imo. You can put whatever code you want under the test source set (generated or not). The kotlin.test infra is heavily reliant on Gradle so indeed if you reuse that you're going to fall into the same trap unfortunately. Could TestBalloon offer some customization options for that 
main
entry point? I could customize the Gradle task depending on a project parameter and have it called 
run
for example. Stuff like that.
o
I'd like to understand why it matters: If some external code can call 
run
, why can't it call 
main
?
e
Because it's an hardcoded/established rule.
o
And you can't introduce a shim into that JS bundle where 
run
just calls 
main
?
e
I can, sure. But is that main function visible in Kotlin code? Otherwise I cannot compile. And more importantly, does it give me access to test results/does it throw when a test fails?
o
IIRC, the shim would probably have to be JS, as this is how it's done in Kotlin's JS test infra (KGP uses a JS script to bootstrap tests). And no, the main function would run tests and report results either via Mocha or via TeamCity, and, optionally, via custom test reports as mentioned above. Test frameworks never throw on failing tests, as they typically run a multitude of tests, with possibly many of them failing. So these are incompatible paradigms.
e
I'm not sure if it can work as a JS shim. Mostly because 
main
might be mangled by the Kotlin compiler. I can definitely use a custom test reporter, but I need to check how it would work as everything should be done at compile time. The best approach would be for the 
run
function to be in Kotlin too, and it being able to call 
main
.
Now that I think about it, 
main
would be executed at JS module load. That's a big issue because it means as soon the test harness loads the file, tests run, instead of waiting for a 
run
call.
o
I mean, it’s open source. You could experiment with a local build and see what works for you.
e
Yup that seems the way to go. I'll give it a try soon.
👍 1
The TLDR of this discussion would probably be: the test framework should work without the build system running it.
18 Views
Open in Slack
PreviousNext
Powered by