Hi all! I'm writing an app and adding screenshot tests for my components and I'm facing some issues ...

# compose

[Pedro Gomez]

Hi all! I'm writing an app and adding screenshot tests for my components and I'm facing some issues with the RoundedCornerShape. When using a shape for the component background the rendering result is not the changes a little bit and this makes the screenshot test tool to report there is an error in the rendering. The shape I'm using is this

RoundedCornerShape(size = 16.dp)

The error reported by the screenshot testing shows a bunch of small different pixels making the tests fail. Red dots in the screenshot are the errors reported by the testing tool. Do you know if the rendering can vary depending on the build or any other param?

[Andrey Kulikov]

could you show the code change which caused such diff in screenshots? or what you are saying is there were no changes?

[jim]

Specifically, I'm not sure how to parse this sentence "When using a shape for the component background the rendering result is not the changes a little bit..."

[Pedro Gomez]

the code is the same, it's just a box with a rounded corner Shape

@Composable
fun Skeleton(color: Color = MaterialTheme.colors.surface) {
    Box(Modifier.background(color = color, shape = RoundedCornerShape(size = 18.dp)).fillMaxSize())
}

The only difference is where the tests are being executed. In my laptop or the GitHub actions.

Sorry @jim, part of the sentence is missing. What I mean is there are no code changes between the execution of this test in local and the CI environment. However, as you can see in the screenshot, the rendering result is not the same. The shape changes a little bit.

[Pedro Gomez]

@ppvi this is the screenshot I mentioned by DM

[ppvi]

GitHub actions use a very small screen for tests, how are you checking the emulator params?

[Pedro Gomez]

the emulator is the same in CI and my local environment. I use a script like this to create the emulator: https://github.com/Karumi/Shot/blob/master/shot-consumer/scripts/create_emulator.sh and one like this to start it https://github.com/Karumi/Shot/blob/master/shot-consumer/scripts/start_emulator.sh

[Pedro Gomez]

the emulator configuration is the same https://github.com/Karumi/Shot/blob/master/shot-consumer/scripts/config.ini

[Pedro Gomez]

I think I'm going to create a branch reproducing the error. it's going to be easier xD

[ppvi]

does Shot compare screenshots even if they have different sizes? or would that throw?

[Pedro Gomez]

nope. if the screenshot has a different size Shot will fail with a different message. It sais the screenshots size should match before comparing them

[ppvi]

does this happen with every single test you run on CI?

[Pedro Gomez]

yes. I think I can get a working example soon

[Pedro Gomez]

@cb this is the interesting issue I've found 🙂

[Pedro Gomez]

another example

[ppvi]

Can you downgrade to whatever compose alpha you used to generate the golden images and check?

[ppvi]

although the fact that it passes locally is strange

[ppvi]

can you link the PR?

[Pedro Gomez]

ready @ppvi I was working on it.

I've created a PR with a simple component you can find here: https://github.com/Karumi/Shot/blob/compose-reproducible-rendering-issue/shot-consumer-compose/app/src/main/java/com/karumi/shotconsumercompose/MainActivity.kt#L52. Once the component was ready, I've created a screenshot test for this component https://github.com/Karumi/Shot/blob/compose-reproducible-rendering-issue/shot-consumer-compose/app/src/androidTest/java/com/karumi/shotconsumercompose/RoundedCornersBoxScreenshotTest.kt and pushed the code so we can see CI failing. You can find the CI build failure here: https://github.com/Karumi/Shot/runs/1252791041?check_suite_focus=true If you download the artifact named "Reports" from the github action you'll find the HTML report with the images inside. You'll need to use a tool to diff images because the difference is so small you will not notice it without the tool. I'm going to attach the diff here anyway:

[jim]

Seems like a graphics-level difference. Maybe @Nader Jawad or @romainguy would have an idea? Also cc @Filip Pavlis in case he has seen something similar with golden files.

[jim]

Specifically, I'm speculating that maybe there is some hardware-accelerated graphics involved and so platform differences might result in subtle pixel differences? But I know nothing about how that works, so maybe I'm just talking nonsense.

[ppvi]

that's my speculation as well but these are emulators! Do they delegate rendering to the underlying OS somehow?

[Pedro Gomez]

🤔 let's see what they say

[jim]

@Pedro Gomez Perhaps try playing with the -gpu <mode> as described here: https://developer.android.com/studio/run/emulator-acceleration#command-gpu

[jim]

Specifically, I'd speculate that running in software-mode would be more consistent than auto or a hardware mode.

[Pedro Gomez]

@jim what's the mode you want me to try? the choices are : auto, host, swiftshader_indirect, angle_indirect, and guest

[ppvi]

I'd start with guest

[Andrey Kulikov]

what we basically do is just a regular Android's canvas.drawRoundRect with antialised Paint . So the issue should be not even Compose specific. I guess different emulators you use do antialising differently

[Pedro Gomez]

🤔 @Andrey Kulikov the emulators configured are the same, the only thing changing is the environment. In local I use a mac book pro and CI uses github actions with a macos image

[Pedro Gomez]

gpu mode = guest it does not work 😞 @Andrey Kulikov is there any chance the team who develop the method ``captureToBitmap`` in the compose testing library is able to provide a version w/o antialising?

[ppvi]

does compose-reproducible-rendering-issue need a `repo`folder with the snapshot artifact?

[Pedro Gomez]

if you run ./gradlew uploadArchives from the root folder it will generate the repo folder for you.

[Pedro Gomez]

@ppvi look at this build! https://github.com/Karumi/Shot/runs/1253306441?check_suite_focus=true looks like the gpu mode configured as guest is working

[ppvi]

after updating it to AGP alpha13 and compose alpha04 the test failed on Linux

[ppvi]

so it might be a Linux/mac thing. We're seeing the same behavior in the Rally sample cc @Manuel Vivo

[ppvi]

spoke too soon, the error is <http://java.io|java.io>.IOException: No such file or directory

[Pedro Gomez]

@ppvi do you know if it is expected to see a black screen emulator when you start it in guest mode?

[ppvi]

no idea

[romainguy]

Different GPUs and different CPUs might introduce small differences in rendering. You never want to do exact pixel to pixel comparisons, but use some kind of difference threshold (our cts tests are great examples of that)

[romainguy]

And it's not antialiasing specific. It will happen with gradients, etc.

[ppvi]

so emulators in mac vs linux can render different things?

[romainguy]

Technically even on Linux vs Linux

[romainguy]

And technically even on a single emulator you may get different results depending on interpreted vs JIT vs AOT (yay floats)

[ppvi]

so what do you think about starting the emulator in gpu mode = guest?

[romainguy]

In CI it’s probably what you want anyway because host requires a physical GPU

[romainguy]

I believe swiftshader_indirect is a better solution than guest though

[romainguy]

But no matter what you do, don’t do exact pixel comparisons

[romainguy]

It can fail for sooo many reasons (dithering for instance)

[nickbutcher]

Are the emulators running on the same API level? As Compose uses skia to render and the ska version varies on API levels you can get small rendering issues like this.

We have the same issue in VectorDrawable tests and have to add new golden images on different API levels.

[ppvi]

yeah Pedro confirmed it's exactly the same emulator because it's launched with a script

[Pedro Gomez]

As a temporal workaroudn we've implemented tolerance in our images diffing algorithm so we consider both screenshots are correct if 99% of the pixels are equals. Thank you all for your time!!!

[nickbutcher]

We do something similar in VectorDrawable tests: https://cs.android.com/androidx/platform/frameworks/support/+/androidx-master-dev:vectordrawable/vectordrawable/src/androidTest/java/androidx/vectordrawable/graphics/drawable/tests/VectorDrawableTest.java;l=402

[romainguy]

We do this in all graphics CTS tests of the platform

[Pedro Gomez]

Just to let you know, implemeting tolerance in our diffing algorithm seems to be the best solution. The rendering diff shows that around 0.4% of the pixels can be different depending on where the tests are executed and the UI implemented. I used an app I'm working on as a reference.