Hey everyone! I've been trying to create native E2...

# compose

[fal]

Hey everyone! I've been trying to create native E2E tests for my app (and avoid Appium). To be clear, what I mean by this is my test should theoretically know as little as possible about the app. Of course, under the hood it still knows about the test tags, and it is still instrumentation tests so it is grey-box after all. But, I do E2E test flows, where I always start from the starting activity/screen and go through the regular bootstrap flows etc.

Now what I struggle the most is the fact that I sometimes have to babysit the ComposeTestRule TestClock. So, suppose in my test I have the following:

private val composeTestRule: ComposeTestRule = createEmptyComposeRule()

// and then I launch my starting activity like:
ActivityScenario.launch<T>(startActivityIntent)

We of course need ComposeTestRule to be able to match semantics as it is SemanticsNodeInteractionsProvider . This is fine, but sometimes I will need to either manually advance the composeTestRule.mainClock or use functions like composeTestRule.waitUntil which under the hood advances the clock (if autoAdvance is true). If I don't do this and I have things like the following in my screen:

LaunchedEffect(Unit) {
    delay(1000)
    onEvent(ScreenEvent.CloseThis)
}

it will completely freeze the app and fail assertions. This level of control is great for precise UI tests where I want to specifically care about this, but from the perspective of an E2E test ideally I'd like to not care about this. It's the same when it comes to infinite animations and needing workarounds like withInfiniteAnimationFrameNanos . It feels like I should be able to not care if I don't want to. It seems like if I want to make E2E tests, I should instead do them through UiAutomator2 (which indeed is more black-box), but that has so many tradeoffs on speed and reliability that I feel like there should be a middle-ground no? Maybe I'm just super off-base here, so really curious on any opinions here, thanks!

[Vladimír Púpava]

Would expect that purpose of code with delay is some animation? Something is displayed longer for visual purpose. Would suggest using animation api for this, than in E2E tests running with animations disabled, there will be no delay. That is our setup for E2E tests.

[fal]

Yep it is to delay self-closing in this instance because of some animation. In this instance, I think you're totally right, we should tie it to the animation so when animations are disabled we don't wait at all. I'll definitely do that 👍. The general sentiment though I think remains, we have to be careful of implementation for the sake of our E2E tests and that somehow doesn't feel great

[Louis Pullen-Freilich [G]]

The general sentiment though I think remains, we have to be careful of implementation for the sake of our E2E tests and that somehow doesn't feel great

Not sure I really understand what you mean by this. You need to have some kind of synchronization barrier if the app state needs to settle in a test, and if the app state is driven by something like an animation or something external to the UI, then the test infra needs to be notified / know about it somehow

[fal]

Hey @Louis Pullen-Freilich [G] sorry about the delay, what I mean is, from the perspective of an E2E test, I shouldn't need to care about 'app state settling'. In fact, I am little a bit against idle resources and having to flag in my production code that the app is "idle". It's just incredibly intrusive for the sake of testing. On our tests we simply decided to poll our assertions until a timeout. Maybe it's not as clean has having idle resources on the app itself, but it's way less intrusive. I understand if we disagree though.

But getting back to the example I showed, I just realized we have the same issue on notifications where we have a

val scope = rememberCoroutineScope()

and on this scope we launch a job to dismiss a notification after a delay. For the sake of tests we basically cannot do this, it needs to be a different scope, otherwise I'll have to manually advance the clock everytime we have a notification. And it wouldn't even make sense to be under an animation that gets disabled because here we do want to see the notification and possibly assert against it on the tests.

At the end of the day, I just think that everytime I need to change production code for my tests to work, it's intrusive.

[Louis Pullen-Freilich [G]]

I'm not really sure what else the alternative is though. If there is some background task that is ongoing, and it is unknown to the test environment, how does the test environment know when to run an assertion?

[Louis Pullen-Freilich [G]]

It already waits for idle for things known inside compose, for example existing animations, or recompositions. But for anything external to that, like arbitrary background tasks in your viewmodel / elsewhere, it needs to know somehow. For example by waiting until a certain screen / text becomes visible

[fal]

Ahh, that's why I was saying, for that we decided just to poll assertions, e.g. every 100 ms we check for the assertion until a timeout like 10 seconds to account for asynchronous operations. I realize this is not "optimal" and more flaky from the POV of the test, but it allows our production code to not care about the tests. In fact, that's what any external test framework has to do right? Like with Appium python tests, they need to query repeatedly, they don't synchronize the app with the test code. Again I'm just referring to these E2E tests. On precise UI tests where I need and want control, it's fine and makes total sense, but for these more general integration tests, it just feels a bit wrong. It feels wrong to tell our developers "don't do that!" even though it's valid production code, just because it will break tests, but maybe it's just how it goes