Screenshot Testing for compose Multiplatform Hi :...

# feed

[Ahmed na]

Screenshot Testing for compose Multiplatform

Hi 👋

Published io.telereso.kmp:core-ui-test lib that supports screenshot testing 📸 ,


Setup and usage docs here


• CI friendly, depends on desktop target no emulators/simulators
• Config tolerance
• 40+ screen size (can create custom)
• Custom Animation delay
• Failed Test Report
◦ Deference Image
◦ Onion-skin Compare
• Targets
◦ Desktop (screenshot record/verify)
◦ Android (supported but skipped)
◦ iOS (supported but skipped)
◦ Web (soon)

I find screenshot testing much more reliable than normal UI testing
Also can visualise the changes before merging them,
Especially when working with own design systems


Hope it's helpful 😄

[Stefan Oltmann]

How does the tolerance work?

I deleted all my screenshot tests because I found that way of testing to be not reliable. For some reasons on CI sometimes pixels shifted a bit as if rendering a view wasn’t 100% deterministic. Never understood those weird things, but hate it when CAI builds randomly fail.

[Ahmed na]

@Stefan Oltmann by checking each pixel rgba
If one pixel has crossed the tolerance limit will fail the test ,

this is the core function where verifying happen (colorDifference)

Understand your use case,
This is why we can have the tolerance concept ,
there is no way 2 images will have same pixels on every device
I might do we one extra thing which is customising setting tolerance for each test
So some complex UI might need more relaxed rules than the others

My issue with normal UI testing was the emulators and simulators,
Expensive and hard to maintain without a cloud base solution

[eygraber]

The docs seem to mention that only JVM is supported?

https://kmp.telereso.io/Testing/Screenshot.html#platform-support

[Ahmed na]

@eygraber it supports ios/android too

But the Screen shots will only be taken and verified on jvm ,

I'll enhance the docs

[Stefan Oltmann]

Yes, in my tests I had to deal with the color difference, too. I remember.

But this is not the only thing that can happen. Sometimes pixels had an offset and were shifted be one.
No idea why. Rerun -> pass, return -> pass, return -> fail, rerun -> pass.

I believe that screenshot testing is the only way to reliable tell if something is different, because it's hard to do in code.

Unfortunately I found that the tech is not reliable for unknown reasons.

I guess another approach could be to snapshot the "structure" (Composables with position and size) in a txt file and compare that.
If it does not match, show the screenshots for a reference, but don't use them to do the comparsion.

That's the last idea I had to improve that process, before my boss told me that my time for tinkering with screenshot testing is up and made me remove all that stuff. I wasted two weeks here.

[Ahmed na]

Snapshots are cool too,
ReactNative does that,

Do you know know where compose might have that logic (any lead i'll try to look into it) ?

[Stefan Oltmann]

Unfortunately not. I guess you need to ask that in #CJLTWPH7S

[Michael Paus]

I once, in the old Java/JavaFX days, had a similar problem with screenshot tests failing randomly. In the end it turned out that it was all my own fault. My rendering code iterated over a Map and as we all know, in Java a map does not have a defined iteration order and thus the drawing was not always exactly the same 🧐.

[Stefan Oltmann]

If it's backed by a HashMap, yes. LinkedHashMaps should have this well defined.

[Stefan Oltmann]

Some thing I noticed back then was that CPU rendering was much more deterministic than GPU rendering.

[Stefan Oltmann]

I guess that's just not intended to be compared pixel by pixel. I don't know why it's so difficult.

[Michael Paus]

Switching to a LinkedHashMap was indeed the solution back then.

[Stefan Oltmann]

Yeah, that's a tricky part of HashMap & HashSet: Their order is not defined, so it depends on the way the hashing is done. This may change from platform to platform, vendor to vendor and even between versions. If you stay on the same version on the same platform the order should be always the same.
I usually catch those errors by running on CI, which is a different system.

[Stefan Oltmann]

And of course this helps a lot, too... 😉

[Stefan Oltmann]

LinkedHashMaps iterate on insertion order, which should be deterministic.