The docs are great :+1::skin-tone-3:, very well ex...
# testballoon
d
The docs are great 👍🏼, very well explained and clear!
thank you color 1
🚀 2
Is this maybe a typo, though?
Suspending code🔗
During execution, code can suspend in (
blue code of) // should be green code, no?
thank you color 1
✅ 1
o
You're right! Will fix this.
d
Also, I didn't see any mention of Robolectric... it's still not supported? (Although I'm excited about the junit rule part... thanks!)
o
I still need to look into Roboelectric integration, so no, for this release it's not there.
d
Thanks for everything so far! The only thing that I think we discussed before was about textFixture not being reset per test... what's your current recommendation for that? I don't really like junit's lateinit var and setup() to re-initialize pattern. I'm not even sure there IS beforeTest somewhere except for in the TestConfig.XXX lambdas that might not fit this pattern so well?
In 90% of the cases, I want a stateful fixture to be reset per test and not per test suite...
o
It's described in the How-to: Supply fresh state to multiple tests
d
That would force me to create a new Fixture class for each set of dependencies I need, say a FakePlayer and a PlayerService that takes it, I need both to be reset, and I need to refer to both...
That's also a custom function for each occurance of such a common pattern... and it would also loose whatever's in the TestContext itself...
I mean it could always be generic... but 🤷🏼‍♂️, maybe this is too common a case for this to be copy-pasted into each project.
o
Please provide some code examples for me to check. Let's see if we can find a better solution for your use cases.
d
Copy code
class PlayerServiceImplTest {
    lateinit var fakeEngine: FakePlayerEngine
    lateinit var playerService: PlayerServiceImpl

    @OptIn(ExperimentalCoroutinesApi::class)
    @BeforeEach
    fun setup() {
        Dispatchers.setMain(Dispatchers.Unconfined)
        fakeEngine = FakePlayerEngine()
        playerService = PlayerServiceImpl(fakeEngine)
    }

    @Test
    fun `setting new items to queue should overwrite the current queue`() = runTest {
        playerService.currentQueue.test {
            assertEquals(PlayerQueue.EMPTY, awaitItem())
            playerService.setQueue(streams)
            assertEquals(PlayerQueue(streams, 0), awaitItem())
            playerService.setQueue(streams2)
            assertEquals(PlayerQueue(streams2, 0), awaitItem())
        }
        assertEquals(streams2, fakeEngine._currentStreams)
    }
...
Is this enough? The FakePlayer needs to be reset every time, and fed to the PlayerService constructor (the SUT), and the fake used for assertions in the tests, they are both stateful classes, so I can't have shared state for each test.
o
Yes, that's sufficient, thanks. I'll get back later today.
👍🏼 1
The proposed TestBalloon suggestion would be like:
Copy code
@OptIn(TestBalloonExperimentalApi::class) // for TestConfig.mainDispatcher
val PlayerServiceImplTestBalloonTest by testSuite(
    testConfig = TestConfig.mainDispatcher(Dispatchers.Unconfined)
) {
    class Context {
        val fakeEngine = FakePlayerEngine()
        val playerService = PlayerServiceImpl(fakeEngine)
    }

    @TestRegistering
    fun test(name: String, action: suspend Context.() -> Unit) =
        this.test(name) { Context().action() }

    test("setting new items to queue should overwrite the current queue") {
        playerService.currentQueue.test {
            assertEquals(PlayerQueue.EMPTY, awaitItem())
            playerService.setQueue(streams)
            assertEquals(PlayerQueue(streams, 0), awaitItem())
            playerService.setQueue(streams2)
            assertEquals(PlayerQueue(streams2, 0), awaitItem())
        }
        assertEquals(streams2, fakeEngine._currentStreams)
    }
}
So there's no "resetting", no re-instantiation magic behind the scenes. Just initializing the
Context
in plain Kotlin. And you can literally see that it's done for each test. What do you think?
d
It's still a bit of boilerplate... also it's not as expressive as textFixture { } syntax... you see the wiring code at the top of the tests instead of a more expressive domain language.
o
That's indeed a tradeoff being made here. Why is
testFixture
different and how does it relate to the above? • The
testFixture
is scoped to its suite and initialized only once. Multiple calls return the same value, so it's easy to reason about it. • If you'd like a
testFixture
to create a suite-specific context, you'd also have to define a class. • Of course, you can have multiple `testFixture`s, but then you'd have to remember invoking each of them. I have indeed considered a
testFixtureForEach
, but the downside would be the magic behind the scenes. It's a stealth setup with side-effects that no longer communicates accurately what happens: • Inside a test, the first invocation of
testFixtureForEach()
would create a new object. • Afterwards, in the same test, each invocation of
testFixtureForEach()
would return the same object. So in that respect, I agree with what's excellently explained in detail in The Quality of Test:
If brevity compromises accurate readability, then you must throw the brevity out.
Does that make sense?
b
We've partially worked around this missing feature by using
aroundEach
. Not super clean. The next part is all very brainstormy, BUT: Given TestBalloon's flexibility, you could probably add some magic with refied generics as an extension on a generator function and magically plug that into your test definitions via context parameters, so the generator gets called automagically AND you don't have to explicitly specify it (thanks to context parameters, if you've declared it before your first test)
d
Do you have an example @Bernd Prünster?
b
Copy code
context(suite: TestSuite)
inline fun <reified T> (suspend () -> T).Test(
    name: String,
    testConfig: TestConfig = TestConfig,
    crossinline nested: suspend TestExecutionScope.(T) -> Unit
) {
    suite.test(name, testConfig = testConfig) { nested(this@Test()) }
}


context(suite: TestSuite)
inline fun <reified T> (() -> T).Test(
    name: String,
    testConfig: TestConfig = TestConfig,
    crossinline nested: suspend TestExecutionScope.(T) -> Unit
) {
    suite.test(name, testConfig = testConfig) { nested(this@Test()) }
}


val aGeneratingSuite by testSuite {
    { Random.nextBytes(32).toHexString() }.apply {
        listOf(1, 2, 3, 4, 5, 6, 7, 8, 9, 10).forEach {
            Test("$it") { freshFixture ->
                //will always fail with fresh randomness for each invocation
                freshFixture shouldBe "Hello"
            }
        }
    }
}
FYI, but I feel this can be done better
d
I still think testFixture is really testSuiteFixture... and there could still be room for testFixture which would be per test. I understand that there might be a little magic going on... but even the fact that there's blue and green areas is a bit magic and this is really coming to make up for this. Meaning even blue areas can have fixtures declared with these helpers in test or suite scope. As for the multiple fixture problem, one can only use a fixture in a test scope anyways so one fixture can use another's instance and they'd all be attached to either unique by test or by suite. This keeps all actual usages in the green area, and promotes to more readable tests... it's not just a question of brevity. As for your idea Bernd, it's a nice trick, but it seems a bit less expressive. Someone looking at the test isn't looking at a high level dsl, but rather more of a clever programming trick.
b
Before we were using
aroundEach
, but that only worked by using shared state, which is something you really don't want. Essentially, ye olde JUnit4 test style that prevents concurrency and leaks state all over the place. I'll be sugar-coating my experiment a litte bit to make the keywords a bit more expressive. In the end though: as long as your generator function does not introduce side effects you can then test concurrently, you can nest arbitrarily and the scope is clear. It essentially takes what was boilerplate in Oliver's sketch, and generalises it to make it re-usable. But yeah, it's a programming trick. Or rather: TestBalloon allows for such patterns to emerge. And what more could you want from a solid foundation for multiplatform testing?! For us, at least, migrating from Kotest was transformative and unearthed some really nasty bugs that slipped through the old test suite. As for green and blue being magic: Not having that distinction is a far bigger problem, if you think about what that would mean for nesting tests and running only specific tests some layers down. Just does not compute…
@dave08 I think I got the API figured out. What's your take on this?
d
That could actually be better... but I have to think about this a bit more. I'm not sure I care as much about putting the fixture in as a receiver of the test as actually being able to declare a bunch of used dependencies (with or without a fixture class), that get reset per test... a bit like the pattern above, but avoiding the lateinit and setup block. testFixture combines those into one for a suite, but not per test.
I guess the Prepared library does all this... but if test balloon is so close to it's syntax, and I don't need the extra bells and whistles of it, then I'd rather have one dependency less.
b
Yup, prepared does that for you. My take on that gap was basically: how few lines of code are needed to get there without polluting a suite with global state
o
As you can see, I was very aware of Prepared when considering various designs for TestBalloon, and Ivan helped me understand Prepared's design. I still consider
testFixtures
to be accurately readable as they are scoped to their declaration and retain their value. Test-scoped fixtures are not: They would be accessible within the scope they are declared in, but change their value between tests, but not inside a test. You have to learn this behavior, it's not clear from reading the code. As mentioned above, tests should accurately communicate what they do. So we have different priorities here, and it's not carved in stone. If many people would prefer that style, I can always add a test-scoped fixture later. But I don't want to rush and accidentally offer a risky API that might lead to unhealthy patterns in user code. If you want it now, you can always build your own extension for it or just use Prepared. TestBalloon invites to to pick your own approach, as it is really easy to extend and adapt to your needs. And in addition, Prepare builds on TestBalloon, so you can cherry-pick what works best for you. I think that's a good thing that TestBalloon users are always free to choose what works best for them.
d
How could I build such a function with test balloon @Oliver.O?
I read the article, it's very interesting. But I'm still convinced that people are so used to this pattern in junit that it's pretty much expected. Getting into the blue-green area testing is newer for people (even when I used kotest, I mostly used the one class instance per test mode and declared my vals inside the suite blocks (wherever a state applied to a whole suite), but I still expected it to reset after each test. In junit, there's the setup function, whereas in test balloon you have to create a custom test function and use it all around to have the before behavior, making it less obvious to create such reinitialization blocks.
b
If you want global mutable state shared, you can still create vars, mutate them in
aroundEach
and enforce sequential test execution. granted, three LoC more than JUnit, but it behaves the same, including all the pitfalls. We use that sill for tests we were to lazy to properly migrate with no issues
d
you mean aroundAll?
b
d
I tried reading it's kdocs, it doesn't seem to obvious how to use it?
Oh... that's not too obvious when coming from other frameworks... but yeah, it should do part of the trick.
b
but, tbh, this pattern is at the very top of things that i want to get rid of. its a footgun the size of a bazooka
🤷🏼‍♂️ 1
d
Ideally, I'd have something like Prepared that does it for me (reset logic inside the actual declaration), since we have a dsl and not just regular junit.
b
I am still not quite catching your issue with my take on things. Not that i want to push you towards anything, I want to understand what I don't see. After all: it gives you a fresh fixture for each test, its type-safe, scoped explicitly, shares no global state (unless you explicitly let it) and thus works for concurrent test execution
We've also identified gaps we needed to fill before migrating from Kotest. So we created shims (Oliver helped) and now with TestBallon 0.7 out, I'm externalising them for reusability
d
Your approach first requires context parameters (which I don't have enabled yet on all our projects), and second makes you create a fixture class for each set of components needed.
b
Both valid points. Thanks!
d
Which isn't a big deal, but I think in a bunch of cases it's just clearer to have a few vals at the beginning of the test.
Or inside different testSuite blocks in the hierarchy of tests (if the val isn't needed in all the suites in the main suite)
Or if it should have different values or initial state in each sub-suite block
o
people are so used to this pattern in junit that it's pretty much expected.
I am aware of this. But sometimes you have to say good by to dinosaur testing patterns if they do more harm than good. Yes, the class+helper costs a few lines of code. But these amortize among lots of tests, and your plain Kotlin skills tell you what's going on.
Getting into the blue-green area
That's not specific to TestBalloon or testing (I just tried to make people aware of the situation): It's what happens in languages with higher-order functions. You pass a function that's called later. It's everywhere in Kotlin and almost every modern language, so it's pretty much unavoidable magic that makes us way more productive.
even when I used kotest, I mostly used the one class instance per test mode and declared my vals inside the suite blocks (wherever a state applied to a whole suite), but I still expected it to reset after each test.
Yes, Kotest provided that with
InstancePerTest
isolation mode. And learned the hard way that it brought lots of complications with their DSL and never worked correctly across use cases. So now, Kotest deprecates
InstancePerTest
and directs users to use
InstancePerRoot
, which is even less intuitive, and everything is still JVM-only. So that's a mess with footguns of various degrees. If you look at the number of Kotest issues dealing with when
before*
and
after*
blocks get called, you might imagine that it's something TestBalloon doesn't want to replicate. So yes, you have to learn a new pattern to be safe and multiplatform. Sometimes we just have to, otherwise we all would still be writing COBOL (and never achieve what we can do today).
d
I understand your point of view on this, it's a bit of a grey area, so I understand you hesitating to integrate this into the library, but for those that DO want this pattern, how would they do it?
I'll maybe try reconsidering meanwhile, but for now, there's enough changes involved in migrating to such a library, so I would rather keep a similar structure for now, and just improve on it if possible.
o
Wouldn't Prepared be the easiest option for now?
d
So far, I haven't even managed to get test balloon's tests to be picked up... I'm really afraid to add new libraries which might introduce more requirements and make the setup even more complicated.
But I do need the minimal amount of tools to be able to start writing tests, at least similar to the way I've been writing them until now. Prepared has a bunch of useful features, but I don't need them for this particular project.
o
Just create a small self-contained reproducer and I'm sure we'll get things up and running.
d
This is a big project that was in the hands of not-so-experienced programmers before I started managing it, and it might be quite hard to create such a reproducer in a small amount of time (at least with all the other elements in the build that might be affecting this...) 😵‍💫. I'd have to see if I'll have time to do it. I was kinda hoping with 0.7.1 it would work out of the box, but 🤷🏼‍♂️.
o
Another option you could try: 1. Always make sure the TestBalloon Gradle plugin is present. 2. In the build script, add the following block:
Copy code
testBalloon {
    @OptIn(TestBalloonInternalApi::class)
    debugLevel = DebugLevel.DISCOVERY
}
Then you should see something like this as compilation warnings:
Copy code
w: ...build/generated/testBalloon/src/commonTest/kotlin/EntryPointAnchor.kt:1:1 Plugin de.infix.testBalloon: [DEBUG] Analyzing source file
w: ...src/commonTest/kotlin/com/example/MyFirstTestSuite.kt:1:1 Plugin de.infix.testBalloon: [DEBUG] Analyzing source file
w: ...src/commonTest/kotlin/com/example/MyFirstTestSuite.kt:6:1 Plugin de.infix.testBalloon: [DEBUG] Found test discoverable 'com.example.MyFirstTestSuite'
d
w: Language version 2.3 is experimental, there are no backwards compatibility guarantees for new language and library features w: Plugin de.infix.testBalloon: [DEBUG] Plugin version 0.7.1-K2.2.21 is processing module <player_hostTest>. w: file///Users/david/Documents/.../player/build/generated/testBalloon/src/commonTest/kotlin/EntryPointAnchor.kt1:1 Plugin de.infix.testBalloon: [DEBUG] Analyzing source file w: file///Users/david/Documents/.../player/src/androidHostTest/kotlin/com/.../player/FakePlayerEngine.kt1:1 Plugin de.infix.testBalloon: [DEBUG] Analyzing source file w: file///Users/david/Documents/.../player/src/androidHostTest/kotlin/com/.../player/PlayerServiceTest.kt1:1 Plugin de.infix.testBalloon: [DEBUG] Analyzing source file w: file///Users/david/Documents/.../player/src/androidHostTest/kotlin/com/.../player/PlayerServiceTest.kt435:1 Plugin de.infix.testBalloon: [DEBUG] Found test discoverable 'com.....player.PlayerServiceExtraFeaturesTest' w: file///Users/david/Documents/.../player/src/androidHostTest/kotlin/com/.../player/internal/PlayerEngineTest.kt1:1 Plugin de.infix.testBalloon: [DEBUG] Analyzing source file
It runs all my tests in that module except for the test balloon test it "discovered".
o
So does it run
PlayerServiceExtraFeaturesTest
?
d
no, that's the test ballon test
o
Which Gradle task do you use to run tests?
d
"playercleanTestAndroidHostTest" "playertestAndroidHostTest" is the command and env vars: TESTBALLOON_INCLUDE_PATTERNS=com.....player.PlayerServiceExtraFeaturesTest↘️* It's what intellij gave me when using the >> in the gutter on the test balloon suite
o
That's correct. So your other tests are JUnit 4 I suppose.
Which Android Gradle plugin do you use (exact name and version)?
d
yes, with junit5 vintage
Copy code
testBalloon = { id = "de.infix.testBalloon", version.ref = "testBalloon" }
testBalloon = "0.7.1-K2.2.21"
o
There's no JUnit 5 on Android host tests, right?
d
agp I'm looking now
Copy code
android-kotlin-multiplatform-library = { id = "com.android.kotlin.multiplatform.library", version.ref = "agp" }
agp = "8.12.3"
junit5 vintage works (which is junit5's junit 4 adapter)
I'm using kotlin test annotations
Copy code
getByName("androidHostTest") {
    dependencies {
        implementation(libs.androidx.junit.ktx)
        if (project.findProperty("local.androidHostTestDance") != "removeDependency") {
            implementation(libs.testBalloon.core)

            implementation(libs.turbine)
            implementation(libs.robolectric)
            implementation(libs.junit.vintage.engine)
            implementation(kotlin("test"))
            implementation(libs.androidx.test.core)
            implementation(libs.mokkery.core)
        }
    }
}
o
That host test dance is no longer necessary. You can safely remove it. And I've probably found the culprit: For Android host tests, you have to use TestBalloon's "-jvm" artifact as a dependency, not the multiplatform artifact. See here. Does that work?
d
yeah that works 🎉, thanks! But it still runs all my tests instead of just the suite I selected in the gutter ("all my tests" are still junit...).
I added that one suite with one test balloon test, and ran it from the gutter
Maybe test balloon only filters out from it's own tests?
o
Yes, in that respect there are possibly compromises to be made (selecting individual tests with multiple frameworks in the same module). You could experiment with adding a
--tests
parameter and supplying TestBalloon's test element path (what's now in
TESTBALLOON_INCLUDE_PATTERNS
). You can safely replace the south-west arrow with a pipe
|
and see how that goes. If that doesn't work, try a star
*
right after the top-level suite name. But I'm not sure we can make every combination work: Gradle is just too opinionated, and has trouble with non class/method-based organizations. This is even true for JUnit 5 itself: It can do multi-level hierarchies, but Gradle's reporting cannot.
In your case, maybe this could help:
Copy code
tasks.withType<Test>().configureEach {
    useJUnitPlatform {
        if (System.getenv("TESTBALLOON_INCLUDE_PATTERNS") != null) {
            excludeEngines("junit-vintage")
        }
    }
}
Does it?
d
The only difference that made was to not run a test class that's not in the same kt file as the test balloon test. But the class inside it's file was run.
Even when I put it in a different file it still ran PlayerServiceImplTest (file name PlayerServiceTest)...