I’m sharing a two-part blog series titled '*Automa...

# feed

[Isaac]

I’m sharing a two-part blog series titled '*Automating UI Change Verification with Android Compose Screenshot Testing.*'. Part 1 covers Compose Screenshot Testing. Part 2 explains how to automate this testing using GitHub Actions. I hope this series will be helpful for those considering screenshot testing!

[Stefan Oltmann]

Been there, done that.
It's just not reliable.

[agrosner]

They’ve straddled the line of reliability but overall I’d call them a win on a large team with many connected pieces. They run fast, catch issues, enable you to refactor composables with confidence the output won’t break. Also provide a record of the different expected states. It depends on processes and teams, but keep them small, focused on the major visual states.

[Stefan Oltmann]

I did this for over a year with Ashampoo Photo Organizer and eventually removed them, because they were failing occasionally. Sometimes pixels are different for some reason. Screenshot testing is not reliable. It’s a great idea and I invested a lot of time there, but sadly I had to see that rendering is not 100% deterministic. It works a lot better if you opt for software rendering, but also that doesn’t always work.

[Stefan Oltmann]

I thought I just drop a word of warning here, but that should stop nobody from going that route. 😄

Instead of comparing pixels a way to compare the structure would work better.

[Stefan Oltmann]

Unit tests that randomly fail and break your build when you are on a deadline don’t help.
A flaky test is useless.

[Albert Chang]

We've been using Paparazzi for years and it works really well. Not flaky at all.

[Stefan Oltmann]

Interesting. Unfortunately Paparazzi is Android only.

In the far past I had troubles with Swing components rendering differently on each system and windows version. We needed to keep CI server and dev machines on the same version.

For my app I used a solution based on androidx.compose.ui.ImageComposeScene …
That’s definitely flaky.

[agrosner]

yeah Paparazzi is one that takes a small percentage (0.5%) of drift allowed where it wont fail. thus reducing the likelihood of pixel issues. different JVMs, versions, layoutlibs, etc all have small variances. so far we’ve had mostly nothing but good use of paparazzi (outside of occasional layout bugs like 1 pixel width but massive height components)

[Stefan Oltmann]

It’s possible that Paparazzi took a lot of internal measures to combat flakiness.
Maybe the system described in the blog post has also evolved over what I used back then.
Time will tell.

[Stefan Oltmann]

I believe that a source of variances is the graphics card. How it does render is not deterministic. I noticed a huge improvement after switching to software rendering.

[Stefan Oltmann]

Another issue I had to combat is how long to wait until all rendering is really complete and how to check that. I failed at that. If paparazzi does it well I guess people smarter than me managed to handle all of that.

I just wanted to say that screenshot testing is hard. Hopefully something like paparazzi will be available one day for Compose for Desktop.