I have a pretty simple Android application built w...

# android

[Per Jansson]

I have a pretty simple Android application built with Jetpack Compose, and I’m trying to use UIAutomator to perform a “opaque box” testing on it, simply visiting each page via the app menu. It is incredibly flaky and just once has it been able to perform the test to the end (so the selectors do work). I’m a using `testTag`:s, but I’ve also tried finding UI elements using text or description, even by clicking on a certain x, y coordinate, still the test is really flaky.

This is kind of how it looks:

Composable

modifier = Modifier.testTag("HomeButton")

Test

// I've tried different ways of finding objects and different ways of waiting, but nothing seems to make the tests less flaky

@Test
fun verifyAllPages() {
    device.wait(Until.findObject(By.res("HomeButton")), DEFAULT_WAIT_TIME).click()
    device.wait(Until.findObject(By.res("MenuButton")), DEFAULT_WAIT_TIME).click()
    device.wait(Until.findObject(By.res("MenuItem_1")), DEFAULT_WAIT_TIME).click()
    ...
    ...
    ...
}

Questions that I have
1. Is this really how flaky UI Automator test are expected to be, or am I missing something?
2. What alternatives do I have if I want to write a test that visits all, or at least some, of the screens in the app?
3. How are people in general testing the Android apps? Coming from the web world it is common to build high level, “opaque box” tests using e.g. Cypress or Playwright.

[czuckie]

Hi Per,

I'd open by saying that you should make sure that the device has animations turned off, because that helps a lot in general with flaky tests, and I'd imagine even more so with opaque box testing.

In general, if I were developing the application, I'd be writing tests using Espresso for quote-unquote legacy XML android views, and the Compose test rules for testing the newer Compose based layouts.

UIAutomator is a tool I'll pull out only when dealing with applications I don't have control over (e.g. tapping some links or buttons in Google Chrome as part of a oauth journey in a test).

That being said, I have done some testing of 'release like' binaries (i.e. APK files that might find their way onto the Play store) using Appium, which is a server/client kind of setup that uses selenium like APIs to be able to treat a running application as a series of MobileElement instances.

The Appium tests though IN MY EXPERIENCE should only be used very sparingly. On the project I was on, we had test driven the entire project (living the dream) and the main bugs we had in production were around mistakes we made using proguard. So to cover the most important journeys (sign up) we would use Appium to test the proguarded build, albeit it was pointing at a sandbox environment.

[czuckie]

Even using espresso or compose test rule, you can have flaky tests, whereby on your machine a test might be perfectly fine, but then on a remote CI server they are not. Again, IN MY EXPERIENCE this has mostly been caused by execution speed on local device vs remote. Sometimes you'll think you don't need to worry about synchronising background work with your tests, because it's been 100% stable for you. However, when you get to firebase test labs or bitrise etc, you can find that actually you do need fine grain control over the synchronisation.

The majority of flaky tests I've encountered have always been that by the time I went to assert on the state of the view hierarchy, work I was expecting to have taken place, hadn't yet taken place, OR, work I was expecting not to have taken place, had taken place. This leads you down a path of designing your services in such a way that you can have that granular control to say 'ok it's loading now', 'ok it's returned a value now'.

Things like LiveData and Dispatchers add another level of synchronicity you need to worry about, but there are some known ways to deal with these. Things like InstantExecutionTestRule and the runTest test helper that lets you get a controllable Dispatcher instance.

[czuckie]

But to really aim at your specific example now (sorry for the rambling!)

If your app is completely compose, you may want to use the createAndroidComposeRule<YourMainActivity>() and then use the rule.onNode... APIs to find nodes in your application and interact with them.

[Per Jansson]

Thanks a lot @czuckie, since I’m a bit of a newbie in Android land it is really appreciated that you share your personal experience. And what I did not mention in my first post is that I was only planning to have a handful:ish of these “opaque box” test, covering the main flows of the app.

I will verify that animations aren’t the issue, but I will also for sure have a look at the Compose tests rules and see what I can achieve with those. I assume it’s this you’re talking about: https://developer.android.com/jetpack/compose/testing?

[czuckie]

Exactly that! Good luck!

[Per Jansson]

Just want to loop back and say that using Compose test rules was a way smoother ride, at least so far, and I was able to quickly re-create a similar in that fashion instead. 🌟

[czuckie]

Great work! For what it's worth, whilst debugging you might find that the screen looks blank, or not how you'd imagine it looks, but you can force the screen to update by clicking on 'Evaluate expression' and executing composeTestRule.waitForIdle() (Where composeTestRule is the name of your compose test rule field). I think that's it for my ninja tricks.