Testing question (learning about testing composabl...

# compose

[Colton Idle]

Testing question (learning about testing composables)

This may be a silly question but I have a really simple composable.

It's just two Text s. (you could even argue that this shouldn't be a separate compasable)

Either way it's legitimately being used in my app and now I've been watching @ppvi (it says Jose if offline, so hopefully that doesn't ping him. Wasn't my intention.) compose testing talk and I want to write a test, but I've never really tested individual views before (not an android and not in any language) and now I'm not sure what to really assert. Maybe since it's such a contrived example there's not much to test and so it's actually a bad candidate, but should I just test that my composable is there with the text that I passed in the parameter? Any guidance would be appreciated.

[jim]

My favorite/recommended solution would be to make the Composable MPP and run it using DesktopScreenshotTestRule. That should be fast and stable and a generally ideal solution.

Failing that, I know there is some code in the androidx repository around taking screenshot tests on Android emulator, but I'm a little less familiar with that code and I think it's a little slower/flakier.

cc @Filip Pavlis for more thoughts.

[jim]

This is how we do it on desktop: https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:compose/[…]androidx%2Fplatform%2Fframeworks%2Fsupport:compose%2F&start=1

[Colton Idle]

@jim SCreenshot testing would be more for testing a design. Like I've read about people at large companies pixel perfect testing a design (facebook? i think at droidcon talked about it once)

[Colton Idle]

I'm fine with it running on an android device. I guess my question is just that I don't know what to test.

[jim]

Testing UI is notoriously hard. The challenge is that there are so many things that can impact the UI that it's hard to write a test which will break when the UI is doing the wrong thing. Also, there are many things which you can test which wouldn't break the UI/pixels but would fail a structural compare. Thus, screenshots are a really good way of ensuring that the user is seeing what you intended them to see, without testing too much or too little.

[Colton Idle]

Like is this a dumb test

class CustomComposablesTest {
@get:Rule
val rule = createComposeRule()

@Test
fun MyTest(){
rule.setContent{
MyComposable("one", "two")
}
rule.onNodeWithText("one two").assertIsDisplayed()
}
}

[jim]

Yeah, you could do that, but now what if the text is the wrong size? Or the wrong color? or obscured by another element?

[Colton Idle]

Testing UI is notoriously hard. The challenge is that there are so many things that can impact the UI that it's hard to write a test which will break when the UI is doing the wrong thing.

Thanks. Thats reassuring actually.

[Adam Powell]

this is where composables being so flexible leads to different testing strategies depending on what it is you're actually building. If you have a very design-focused composable, screenshot testing is a great idea since that design is the API contract. Something more structural, you might want to test the resulting semantics tree, something more functional/architectural, you would want to test its behavior and when it sends callbacks, etc.

[Colton Idle]

@Adam Powell yeah. In my case, I'm not trying to test the design. I'm more focussed that the two params being passed in actually show. So to your point... I want to test sstructually? so I want to test the resulting tree for a single instance of both args.

[Colton Idle]

Thanks both @Adam Powell and @jim. its nice to know this is "hard" and im not just missing something basic. I just want to set the right precendence for my team and show how easy it is to test.

[jim]

The only way to verify something is showing is with a screenshot, because like I said, there might be another element obscuring it (not visible) the color might be white (not visible), the font might not have loaded (not visible), the size might be too small (not visible), etc.

[jim]

You can maybe some amount of signal from a structural test, certainly better than nothing. I would use structural tests for things which require interaction, like "perform click on button", where you want to grab the button by semantics nodes (obviously you don't want to write an artificial intelligence that analyzes the image to find the button and click it).

[Adam Powell]

right. Sometimes it doesn't matter if something is pixel perfect, and sometimes testing the exact rendering of your dependencies is testing those dependencies more than it's testing your component. It varies quite a bit.

[Adam Powell]

But it'd be really nice if more folks would run tests that make sure text doesn't get cut off when the user increases the systemwide font size 😄

[jim]

If your dependencies change in such a way that it causes your behavior to change, wouldn't you like to know? 😛

[Adam Powell]

Depends on the stability of those dependencies. If it's something like our own Text composables post-1.0, I'd hope I wouldn't need to test for compatibility of that version over version in my own apps 🙂

[Colton Idle]

But it'd be really nice if more folks would run tests that make sure text doesn't get cut off when the user increases the systemwide font size

OoooH. Thats a good freakin test idea. lol

[Colton Idle]

I'm really interested in seeing if compose will provide standard support for screenshot testing. I know Jose said in his talk that theres a sample little library in some repo that is being used for the time being (maybe its what Jim mentioned above).

Anyway. thanks everyone for the input. really interested in seeing where composable tests end up once it has wider adoption and more best practices start to appear

[Colton Idle]

Now this also begs the question of "Should every composable go into a separate file i.e. MyComposable, so that cmd + shift + t in intellij brings you to MyComposableTest by default"

[Colton Idle]

so far, Ive just been throwing all my composables in the same file for simplicity and it still hasn't really fallen over... yet. but its early days

[Adam Powell]

If you remove, "composable" from the statement and say, "should every function go into a separate file?" I think a lot of people would find that strange 🙂

[Adam Powell]

but if that composable is something functionally interesting, with an associated hoisted state type declared in the same file, then it's not so strange

[Adam Powell]

or otherwise one concept with a few overloads/variants alongside it

[Colton Idle]

@Adam Powell good way of thinking about it. Now I'm just having "analysis paralysis" on which dir/package the test actually goes in.

[Filip Pavlis]

I'm not sure I'm on the same page with necessarily using screenshot test. assertIsDisplayed() gives pretty strong guarantees and is cheap to maintain unless something is overlaying your composables in compose which is not that super frequent.

[Filip Pavlis]

For two simple texts you probably don't need screenshots

[Filip Pavlis]

Also there is no need for desktop test. These asserts will do totally fine in Android test.