https://kotlinlang.org logo
p

Pedro Gomez

10/13/2020, 2:17 PM
Hi all! I'm writing an app and adding screenshot tests for my components and I'm facing some issues with the RoundedCornerShape. When using a shape for the component background the rendering result is not the changes a little bit and this makes the screenshot test tool to report there is an error in the rendering. The shape I'm using is this
Copy code
RoundedCornerShape(size = 16.dp)
The error reported by the screenshot testing shows a bunch of small different pixels making the tests fail. Red dots in the screenshot are the errors reported by the testing tool. Do you know if the rendering can vary depending on the build or any other param?
a

Andrey Kulikov

10/13/2020, 3:25 PM
could you show the code change which caused such diff in screenshots? or what you are saying is there were no changes?
j

jim

10/13/2020, 3:47 PM
Specifically, I'm not sure how to parse this sentence "When using a shape for the component background the rendering result is not the changes a little bit..."
p

Pedro Gomez

10/14/2020, 5:56 AM
the code is the same, it's just a box with a rounded corner Shape
Copy code
@Composable
fun Skeleton(color: Color = MaterialTheme.colors.surface) {
    Box(Modifier.background(color = color, shape = RoundedCornerShape(size = 18.dp)).fillMaxSize())
}
The only difference is where the tests are being executed. In my laptop or the GitHub actions. Sorry @jim, part of the sentence is missing. What I mean is there are no code changes between the execution of this test in local and the CI environment. However, as you can see in the screenshot, the rendering result is not the same. The shape changes a little bit.
@ppvi this is the screenshot I mentioned by DM
p

ppvi

10/14/2020, 7:54 AM
GitHub actions use a very small screen for tests, how are you checking the emulator params?
p

Pedro Gomez

10/14/2020, 7:55 AM
the emulator is the same in CI and my local environment. I use a script like this to create the emulator: https://github.com/Karumi/Shot/blob/master/shot-consumer/scripts/create_emulator.sh and one like this to start it https://github.com/Karumi/Shot/blob/master/shot-consumer/scripts/start_emulator.sh
I think I'm going to create a branch reproducing the error. it's going to be easier xD
p

ppvi

10/14/2020, 7:58 AM
does Shot compare screenshots even if they have different sizes? or would that throw?
p

Pedro Gomez

10/14/2020, 7:59 AM
nope. if the screenshot has a different size Shot will fail with a different message. It sais the screenshots size should match before comparing them
p

ppvi

10/14/2020, 8:01 AM
does this happen with every single test you run on CI?
p

Pedro Gomez

10/14/2020, 8:15 AM
yes. I think I can get a working example soon
@cb this is the interesting issue I've found 🙂
another example
p

ppvi

10/14/2020, 10:02 AM
Can you downgrade to whatever compose alpha you used to generate the golden images and check?
although the fact that it passes locally is strange
can you link the PR?
p

Pedro Gomez

10/14/2020, 10:43 AM
ready @ppvi I was working on it. I've created a PR with a simple component you can find here: https://github.com/Karumi/Shot/blob/compose-reproducible-rendering-issue/shot-consumer-compose/app/src/main/java/com/karumi/shotconsumercompose/MainActivity.kt#L52. Once the component was ready, I've created a screenshot test for this component https://github.com/Karumi/Shot/blob/compose-reproducible-rendering-issue/shot-consumer-compose/app/src/androidTest/java/com/karumi/shotconsumercompose/RoundedCornersBoxScreenshotTest.kt and pushed the code so we can see CI failing. You can find the CI build failure here: https://github.com/Karumi/Shot/runs/1252791041?check_suite_focus=true If you download the artifact named "Reports" from the github action you'll find the HTML report with the images inside. You'll need to use a tool to diff images because the difference is so small you will not notice it without the tool. I'm going to attach the diff here anyway:
j

jim

10/14/2020, 10:44 AM
Seems like a graphics-level difference. Maybe @Nader Jawad or @romainguy would have an idea? Also cc @Filip Pavlis in case he has seen something similar with golden files.
Specifically, I'm speculating that maybe there is some hardware-accelerated graphics involved and so platform differences might result in subtle pixel differences? But I know nothing about how that works, so maybe I'm just talking nonsense.
p

ppvi

10/14/2020, 10:50 AM
that's my speculation as well but these are emulators! Do they delegate rendering to the underlying OS somehow?
p

Pedro Gomez

10/14/2020, 10:52 AM
🤔 let's see what they say
j

jim

10/14/2020, 10:57 AM
@Pedro Gomez Perhaps try playing with the
-gpu <mode>
as described here: https://developer.android.com/studio/run/emulator-acceleration#command-gpu
Specifically, I'd speculate that running in software-mode would be more consistent than
auto
or a hardware mode.
p

Pedro Gomez

10/14/2020, 11:24 AM
@jim what's the mode you want me to try? the choices are : auto, host, swiftshader_indirect, angle_indirect, and guest
p

ppvi

10/14/2020, 11:31 AM
I'd start with
guest
a

Andrey Kulikov

10/14/2020, 11:44 AM
what we basically do is just a regular Android's
canvas.drawRoundRect
with antialised
Paint
. So the issue should be not even Compose specific. I guess different emulators you use do antialising differently
p

Pedro Gomez

10/14/2020, 11:48 AM
🤔 @Andrey Kulikov the emulators configured are the same, the only thing changing is the environment. In local I use a mac book pro and CI uses github actions with a macos image
gpu mode = guest it does not work 😞 @Andrey Kulikov is there any chance the team who develop the method ``captureToBitmap`` in the compose testing library is able to provide a version w/o antialising?
p

ppvi

10/14/2020, 12:23 PM
does
compose-reproducible-rendering-issue
need a `repo`folder with the snapshot artifact?
p

Pedro Gomez

10/14/2020, 12:40 PM
if you run ./gradlew uploadArchives from the root folder it will generate the repo folder for you.
@ppvi look at this build! https://github.com/Karumi/Shot/runs/1253306441?check_suite_focus=true looks like the gpu mode configured as guest is working
p

ppvi

10/14/2020, 12:57 PM
after updating it to AGP alpha13 and compose alpha04 the test failed on Linux
so it might be a Linux/mac thing. We're seeing the same behavior in the Rally sample cc @Manuel Vivo
spoke too soon, the error is
<http://java.io|java.io>.IOException: No such file or directory
p

Pedro Gomez

10/14/2020, 1:02 PM
@ppvi do you know if it is expected to see a black screen emulator when you start it in guest mode?
p

ppvi

10/14/2020, 1:15 PM
no idea
r

romainguy

10/14/2020, 2:47 PM
Different GPUs and different CPUs might introduce small differences in rendering. You never want to do exact pixel to pixel comparisons, but use some kind of difference threshold (our cts tests are great examples of that)
And it's not antialiasing specific. It will happen with gradients, etc.
p

ppvi

10/14/2020, 2:48 PM
so emulators in mac vs linux can render different things?
r

romainguy

10/14/2020, 3:25 PM
Technically even on Linux vs Linux
And technically even on a single emulator you may get different results depending on interpreted vs JIT vs AOT (yay floats)
p

ppvi

10/14/2020, 3:26 PM
so what do you think about starting the emulator in gpu mode = guest?
r

romainguy

10/14/2020, 3:27 PM
In CI it’s probably what you want anyway because host requires a physical GPU
I believe
swiftshader_indirect
is a better solution than
guest
though
But no matter what you do, don’t do exact pixel comparisons
It can fail for sooo many reasons (dithering for instance)
n

nickbutcher

10/15/2020, 8:20 AM
Are the emulators running on the same API level? As Compose uses skia to render and the ska version varies on API levels you can get small rendering issues like this. We have the same issue in VectorDrawable tests and have to add new golden images on different API levels.
p

ppvi

10/15/2020, 8:27 AM
yeah Pedro confirmed it's exactly the same emulator because it's launched with a script
p

Pedro Gomez

10/15/2020, 8:54 AM
As a temporal workaroudn we've implemented tolerance in our images diffing algorithm so we consider both screenshots are correct if 99% of the pixels are equals. Thank you all for your time!!!
r

romainguy

10/15/2020, 3:04 PM
We do this in all graphics CTS tests of the platform
p

Pedro Gomez

10/20/2020, 5:29 PM
Just to let you know, implemeting tolerance in our diffing algorithm seems to be the best solution. The rendering diff shows that around 0.4% of the pixels can be different depending on where the tests are executed and the UI implemented. I used an app I'm working on as a reference.
7 Views