Questions for everyone interested in improving Kot...

# android

[Oliver.O]

Questions for everyone interested in improving Kotlin+Android testing (from the author of TestBalloon, a new Kotlin test framework):

If you weren't tied to JUnit 4, how would you like to write Kotlin+Android tests (local, instrumented, Compose, ...)? What are you missing? What's good the way it is? All ideas welcome! 🧵Please.

[trevjones]

IMO the main hurdle with getting away from junit 4 for androidTest is that the vast majority of scalable test device providers are tied to how the android junit runner works.

Meaning as soon as you need more than one or two phone's that can fit on your one CI node you are probably going to have to revert to junit 4 to jump elsewhere.

The other part and likely biggest part is so much of the androidx testing ecosystem assumes junit4 by default so if you want out of that world you are likely walking away from all the tooling they provide.

[Oliver.O]

Thanks for the insights!

To clarify: TestBalloon integrates with JUnit 4 and the Android JUnit runner, so the underlying ecosystem gets everything it expects. In addition, where JUnit 4 fits the bill, you can just keep your tests based on it, they work nicely side by side with TestBalloon.

What can change is the user interface for devs, from 2006 Java style (static, annotated magic, before/after) to 2025 idiomatic Kotlin (dynamic, explicit, DSL, lambdas, concise). Think about generating test variants for edge cases in plain Kotlin with expressive names.

That's why I'm asking: What's just good enough from a DX perspective, what's a real pain today? Which tests don't get written because they are just too cumbersome to get right?

[gildor]

I think standard JUnit test style is definitely not optimal, it works for many cases, but for many it's too cumbersome, JUnit 5 provids many more nice features with test generation.

DX could be improved if Junit 4 APIs wouldn't be expose, like I could use own assert library and Junit wouldn't pollute autocompletely

I would love to see DSL, but only if it will be well supported by build tooling and test runners
For example Spek was really terrible mess, library and initiative is good, but without strong IDE support, stable and clean API and JUnit 5 support on Android it was so painful

Additionally, I really don't like idea to use different test frameworks for Unit and Instrumentation tests, it causes a lot of problems and makes test hard to port from one to another type.

[gildor]

I checked TestBaloon, it look very promising!

But again, I'm really burned by Spek 😄

[Oliver.O]

Thanks for sharing your thoughts!

One clarification: I did not intend to suggest using different frameworks for local and instrumented tests. I mainly wanted to point out that TestBalloon can coexist (both for local and instrumented tests) with legacy JUnit4 tests in the same module, giving you a smooth incremental migration path and the freedom to use choose frameworks according to your needs.

So you'd love to have a DSL-powered framework with a clean and stable API if that is well integrated into the ecosystem. (If so, I'd say TestBalloon already has these characteristics. In particular, a lot of effort has been invested into making it a good citizen in JUnit4/Android local+instrumented and Kotlin Multiplatform worlds (JVM/JUnit5, Native, JS, Wasm), plus Gradle, of course.)

The integration aspect seems to be what dominates so far, and I can totally understand the scars you take away after getting stuck in rabbit holes of frameworks with integration loopholes, incomplete functionality, and/or hard-to-use APIs.

Would it be correct to summarize it this way?

[gildor]

Yep, absolutely
My comment was written before I checked TestBalloon
It's indeed answers pretty much all of my concerns and you summarized it really well

I'm actually eager to try it now

[gildor]

Yeah, I think ide experience and specifically Android test runners for instrumentation are my main points of worry in terms of adoption

[Oliver.O]

Thanks! And really, I feel you. I have been in too many rabbit holes myself, and I'm really hesitant to put something out there for people to try without also trying to explain its limitations (TestBalloon had a separate doc for that initially, and now I'm trying to keep issues current and the issue count low).

[trevjones]

yeah some examples of doing some androidTest UI testing where you use existing JUnit4 style rules would be helpful.

Things like permission granting, compose rule, custom rules to handle DI. etc...

[gildor]

I think a part of the problem with adoption is how the whole java/kotlun community and ecosystem, especially android part of it, used to JUnit 4 style tests and knows no alternative and may not understand that tests not always extremely boring and boiler plate

So teaching devs is a big part of the problem, JUnit was a thing for so long, second generation of coders essentially doesn't know anything else

[gildor]

Even just understand how to organize tests could be challenging, with memorization and lazy execution in mind

I remember how challenging it was for me initially to used to spec style tests, even though I was very interested, I could expect struggle and impatience of devs who could be forced to use completely new style of tests

I think everything is solvable, but the rest of the experience should be really good to help overcome the friction

[Oliver.O]

Excellent aspects! We are all used to frameworks that have not aged very well, e.g. JUnit class/method/magic reinstantiation contrasting with Kotlin coroutines scope requirements, or having to learn a new (restrictive) language for dynamic tests instead of creating them in plain Kotlin.

So these are exactly some of the questions I had before posting here: I am fully aware that TestBalloon users could benefit from more documentation with good test patterns, and this is in the works. I need to find out which patterns would fit devs best, where a modern frameworks boosts dev happiness, and where the old stuff is just good enough. So that's exactly why I was asking. And I'm also trying to find out whether some Compose test integration like the multiplatform `runComposeUiTest` would be worth considering.

It may not be obvious for those accustomed to the old way of doing things, but, as @CLOVIS put it in this article:

Even after JUnit's dominance in the JVM ecosystem, we can still make tests easier to write and maintain. Kotlin makes possible patterns that we couldn't dream of in the Java world.

[David]

Generally I feel the setup with JUnit tests are pretty clunky, one of the major problems we've run into testing on android is flakiness, in e.g E2E tests. It can be system suddenly prompting a dialog update dialog, the API for waiting for a screen to settle has been iffy so maybe a click happens a bit too quickly before the screen is running, etc. To avoid flakiness we've started running our tests nightly about 100 times, so support around this I think can be helpful..

To combat the flakiness we have multiple annotation/test rules/extensions
• Capturing screenshot on test failed.
• Capture screen recording of the entire test.
• Annotations that reads arguments from the instrumentation registry if to run a test or not. E.g:
◦ If we have access to some local network asset or not
◦ To run a test if the build is of a certain flavour (e.g Google Play build).
So I think good support around adding tools like these on different levels (individual tests, group of tests, all tests) could be very beneficial as well.

Also a completely separate idea, I think a channel in kotlinlang (#testballoon?) would be nice. 🙂

[Oliver.O]

Oh, 100 times is a lot. Could you provide a minimal example on what such a flaky test currently looks like? I have some ideas how such situations could improve with TestBalloon, but I'd better not speculate what the problem might look like exactly.

[gildor]

I have an opposite view to David's
I felt that test runner should be stable and flexible by itself, and shouldn't include any additional domain specific things (like helper functions for assume environment), or some pre setup for Android flavours

It's all makes unnecessary bloat, it exists even in simplistic junit, where I don't need their assert library or dependency on hamecrest matchers

Make test framework extendable and allow to build infrastructure around it is many times more important than support some specific use case, it all blurring the focus

Be lean and focused exactly why many square libs are popular, and their competitive libs died, they wanted to please everyone instead of being good on specific thing

[Oliver.O]

I agree with keeping the framework minimal, but extensible. That's the current design and it should stay that way. The two existing extensions (for blocking detection and Kotest Assertions support) are to be separated into their own libraries. I'd do the same for anything Compose.

But given that, it would help to provide some extensions (as separate libraries) to get people up to speed and to establish patterns about how to benefit from extensibility.

Does that make sense?

[David]

@Oliver.O I don't really have an easy reproducible for the flakiness. But there are functions like waitForStableInActiveWindow (doc) that has been added lately to allow the screen to settle before proceeding with a test clicking on a item to help prevent this, one can easily imagine how a test may fail if a item is pressed before the screen actually has settled and are in resumed state. The API itself has also been a bit flaky:
https://issuetracker.google.com/issues/420349130
https://issuetracker.google.com/issues/417046391

Generally I agree with you @gildor, the test framework shouldn't but a bandage over problematic uiautomator behaviour. However, reality to iron out the flakiness of tests in Android one needs to run them multiple times. There may still be other aspect interfering with your tests if you run on an actual device that you don't have control over, e.g a system dialog showing etc.

[David]

To be clear; the flakiness we've seen over the years has been on E2E ui tests, using uiautomator , it has gotten some love now recently with Marcello at Google doing some work, but before that it has been a bit of a forgotten child that is on life support.

[gildor]

And instrumentation test flakiness is a larger problem, runner could help with allowing retry, or priorize flaky tests, but outside of those, I don't think it's runner job

[Oliver.O]

Would something like the `eventually` function in Kotest Assertions help? It wraps a group of assertions, which it then retries up to a time and/or iterations limit.

If so, for my KotlinConf 2024 talk, I was showcasing a version of it which worked independently from the Kotest Assertions library, so it could be used with any assertion library.

[David]

I think the key here I'm getting is extensibility, being able to apply configuration to a test or set of tests, e.g I want all UI tests to take a screenshot if they fail or record their entire run. (The screenshot part should not be part of the test framework, but maybe would be nice if it is a third party plugin/module I can setup?). I also think it can be good if there is an opinionated way of doing this, so not everyone comes up with their own solution.

Regarding the flakiness:
I'm not really sure. Problem with instrumented tests on android is that you may never fully get rid of the flakiness. There may always be a pop-up that the system pushes that will be breaking your logic. I don't think eventually would fully help, and logically the test should fail because it didn't get what it expected. Running it again would launch the app, putting it on top and then test would run fine. 🤷

Something like eventually is what you end up doing, waiting for a object to a appear, hope to find it to then click it:
https://github.com/mullvad/mullvadvpn-app/blob/5621df99a0f357262fef6fe886b2fbf956f[…]llvad/mullvadvpn/test/common/extension/UiAutomatorExtensions.kt

Any resistance again flakiness would have to be done on a higher level, e.g run the test 10 times, expect at least 80% success, or do a retry if a test failed to make sure it wasn't flaky. In the end, I sort of agree with Andrey, flakiness shouldn't be solved by the testing framework, it should be solved at it's core. However, reality is what reality is, ui testing on android device will really never be non-flaky.

[Oliver.O]

If the flakiness occurs at the level of an entire test, sure. I was just asking myself whether that could be stripped down to deal with flakiness at lower levels in order to avoid re-running entire UI setups. But it doesn't really matter: Flakiness exists on devices, emulators and in even backend communications if we include more of the real stack in our E2E tests. So it makes sense to have tools that deal with it.

TestBalloon already provides extensibility at every level of the test hierarchy, so applying something to all kinds of UI tests is easy. I'll do something about it to demonstrate what's currently possible and then we can find out if any further extensibility is required.

Regarding a channel, I can ask Alina to provide one, though it may be a bit early (even the massively promoted #C06V6SFE71D has just 53 users).

[David]

Regarding a channel, I can ask Alina to provide one, though it may be a bit early (even the massively promoted #C06V6SFE71D has just 53 users).

I think it depends on the use case, power-assert is a pretty slim tool. Kotest has 1200+ users. 🎯

[Oliver.O]

So here it is: #C09FQGG85EC 🎉