Hey everyone, I have a question regarding how `Gra...

# compose-android

[Sergey Y.]

Hey everyone,
I have a question regarding how GraphicsLayer works in Jetpack Compose.
I thought I understood it before, but a recent discovery has made me question my understanding 🤔.
Details in the thread 🧵

[Sergey Y.]

Here's the case I don't quite grasp:
I have a simple LazyColumn with a Modifier.drawWithContent that records its content into a GraphicsLayer. Then, I draw this GraphicsLayer in a loop onto an AndroidExternalSurface (backed by a SurfaceView) canvas (see the attached code for reference).

What I've found is that Modifier.drawWithContent is not called every time I scroll the lazy list. This might be due to the lazy nature of LazyColumn and how it reuses the composable widgets under the hood, idk. As a result, the GraphicsLayer only records the initial layout.

However, every time I scroll the list, the continuous drawing loop in my AndroidExternalSurface correctly updates the picture on the canvas, and I'm not sure why. It's puzzling because no new graphicsLayer.record { } is happening during scrolling(logcat is alive).

If anyone has an explanation for this behavior, I would be glad to know. I'm curious to understand how the GraphicsLayer is able to update the canvas without being recorded again during scrolling.
Thanks in advance for any insights!

[Sergey Y.]

GraphicsLayer_mistery.mp4

GraphicsLayer_mistery.mp4

[Sergey Y.]

The code is very simple: https://gist.github.com/desugar-64/06515f38ddb6281d065e8dd2305f0f75

[Zach Klippenstein (he/him) [MOD]]

I thought I understood it before, but a recent discovery has made me question my understanding

This is me every time I think I understand graphics and then talk to Nader.

[Sergey Y.]

I was assuming that to update the GraphicsLayer, I needed to record into it each time the UI updates. It seemed logical to me, and it kind of worked with regular scrollable columns and rows. They invoke Modifier.drawWithSomething each time I interact with them.

However, with the lazy list, it's a different situation. I'm not sure why it works the way it does. I expected that I would need to record into the GraphicsLayer whenever the UI changes, but it seems to update correctly even without explicit recording during scrolling.
🤯

[Zach Klippenstein (he/him) [MOD]]

(i’m not even going to try to answer here, very much out of my wheelhouse)

[Sergey Y.]

Hope this post summons Romain or Nader 😁

[Nader Jawad]

GraphicsLayers are effectively displaylists. When a GraphicsLayer draws another GraphicsLayer instance, there is a reference to the other GraphicsLayer's displaylist. From a mental model standpoint the GraphicsLayer form a DAG. So because of this anytime a GraphicsLayer's properties or displaylist changes (i.e. call to record) the changes would be picked up on the next draw regardless if there's a record or not.

This is one of the reasons why the RenderNode and Picture APIs appear similar, however, are different. This is because RenderNodes (which back GraphicsLayer instances) are effectively pointers, whereas Picture instances do deep copies of the drawing commands

[Sergey Y.]

Thanks for the great explanation, Nader! It really clarifies how GraphicsLayers work under the hood.
Given this knowledge, what would be the typical use cases for GraphicsLayers?
Should I record only once, or is it fine to record on each onDraw call?

[Nader Jawad]

You typically should record once. The overall goal is to avoid re-recording drawing instructions as much as possible. For example, if you have a Column with a background that has a GraphicsLayer for each child, when the child invalidates, the Column should not have to, instead it would just re-draw the existing displaylist that it has which refers to the child Graphicslayer

[Nader Jawad]

It's another reason by design, we exposed access to create + record GraphcisLayers within Modifier.drawWithCache's CacheDrawScope but we don't expose it directly within DrawScope. This is to nudge developers to "do the right thing" by API design

[Sergey Y.]

I was wondering if I could achieve the same result by providing my GraphicsLayer through a simple Layout wrapper, like this:

Layout { measurable ->
    val placeable = // measure
    layout(width, height) {
        placeable.placeWithLayer(x, y, myGraphicsLayer)
    }
}

Instead of using draw callbacks, would this approach work just as well? It seems like it might be a cleaner way to handle the GraphicsLayer without the need for explicit drawing calls.

[Nader Jawad]

placeWithLayer is mostly for authors of Layout composables in order to provide isolation boundaries for child composables.

[Nader Jawad]

If you just want to capture/decorate the rendered result, using Modifier.drawWithCache + obtainGraphicsLayer is preferred

[Sergey Y.]

One more question, if you don't mind! 😊
When I'm drawing onto a hardware-backed canvas for recording into an external OpenGL texture, should I avoid any recordings into the GraphicsLayer during this process?
I want to make sure I'm following the best practices and not introducing any potential synchronization issues. How will it behave in that case?

[Sergey Y.]

it is running on the GL thread.

[Nader Jawad]

Recording is on the UI thread and execution drawing commands is done on the RenderThread. When a render is issued, the UI thread is blocked temporarily to handover the RenderNodes to the RenderThread for recording. So as long as you are recording from the UI thread you should be ok.

[Nader Jawad]

RenderNode has looser threading restrictions than Views, so depending on your API level if you attempt to record on a background thread your results maybe undefined and potentially crash for View backed GraphicsLayers as the View system has strict checks that it is modified only from the UI thread

[Nader Jawad]

If you are using a SurfaceTexture backed Surface to obtain a hardware accelerated canvas, it would be safer to handle all the GraphicsLayer rendering on the UI thread and have your GL code execute on an OnSurfaceTextureAvailableListener

[Sergey Y.]

This is precisely what I've done 🙂

[Sergey Y.]

Otherwise I see some error logs in the logcat.

[Sergey Y.]

image.png

[Sergey Y.]

It works so far.

Screen_recording_20240519_031352.mp4

[Sergey Y.]

I wish I could somehow workaround the issue when a clipped AndroidExternalSurface is flickering.

[Sergey Y.]

surface_view_flicker.mp4

surface_view_flicker.mp4

[Sergey Y.]

https://issuetracker.google.com/issues/341537869

[Sergey Y.]

Sorry for the bugs 😅

[Nader Jawad]

No problem! Thanks for filing and trying out these APIs!

[Sergey Y.]

Is it generally okay to have more than one SurfaceView on the screen from a technical standpoint? In my experience, I've never seen this happen. The typical usage was to have one big view serving as a camera viewfinder or player view.
For my use case, I need more than one SurfaceView/TextureView. I had considered combining the output rendering into a single texture and placing it onto one large background SurfaceView. However, I abandoned this idea because it would require significant changes to the user's layout structure to make it work. Additionally, it might cause further headaches when the blur views are on different zIndex levels.

[Nader Jawad]

Yes it's definitely a supported use case. In fact SurfaceView uses multiple composting layers for is background and main content as well

[Sergey Y.]

I found a workaround for my code.
AndroidExternalSurface only flickers when clipping is applied through Modifier.graphicsLayer, which is used by all of the shadow, clip, and clipToBounds modifiers.
However, when I clip directly via canvas.clip*, it does not flicker!
I'm wondering what the difference is between these two clipping methods 🤔.
I still don't know how layout alignment is involved here, but nevertheless, this fix is suitable for me. Updated the issue information.

Thank you so much for your assistance! :)

[Sergey Y.]

I was wondering, does clipping the Surface-backed canvas result in faster drawing compared to drawing on a full-size canvas?
I'm curious because I'm thinking that clipping the canvas might reduce the area that needs to be rendered, potentially leading to some performance improvements. Am I right, or would the performance boost be negligible?

val hwCanvas = layerSurface.lockHardwareCanvas()
hwCanvas.drawColor(Color.BLACK, PorterDuff.Mode.CLEAR)
drawingScope.draw(density, LayoutDirection.Ltr, Canvas(hwCanvas), sizeDec) {
    clipRect(0f, 0f, areaWidth, areaHeight) {
        drawLayer(graphicsLayer)
    }
}
layerSurface.unlockCanvasAndPost(hwCanvas)

vs

val hwCanvas = layerSurface.lockHardwareCanvas()
hwCanvas.drawColor(Color.BLACK, PorterDuff.Mode.CLEAR)
drawingScope.draw(density, LayoutDirection.Ltr, Canvas(hwCanvas), sizeDec) {
    drawLayer(graphicsLayer)
}
layerSurface.unlockCanvasAndPost(hwCanvas)

[Sergey Y.]

> GraphicsLayers are effectively displaylists. When a GraphicsLayer draws another GraphicsLayer instance, there is a reference to the other GraphicsLayer's displaylist.
Also, I've noticed that GraphicsLayer does not behave like pointers to display lists on Android 6. I must explicitly record them at every drawing call in order to update the texture. I suppose this is because `RenderNode`s became public API only in Android 10.
So, I was wondering, can I safely assume and build my recording logic based on this behavior for SDK levels lower than 29? Or is there a possibility that GraphicsLayer might act like mutable display lists on some other older SDKs?
I noticed that you're trying to bind to the private RenderNode using reflection, but with some exceptions. It made me curious about the consistency of GraphicsLayer's behavior across different Android versions.

[Sergey Y.]

I just discovered a potential reason behind the Surface jumping to (0,0) and back. In the demo, I was using a LazyColumn, and I started noticing that the jumping was occurring at suspicious intervals. The AndroidExternalSurface is not a part of the LazyColumn items; it is drawn above it in a separate composable widget. This led me to think that the LazyColumn might be causing the issue. I replaced it with a plain scrollable Column, keeping everything else the same (including the AndroidExternalSurface with offset), and the flickering disappeared.
Upon further investigation, I found that the flickering (when the surface picture jumps to (0,0) and back) also happens when I add or remove layouts from the composition(and as LazyList does). Therefore, I suppose it occurs during the re-layout of the composable tree, but I have no idea why it jumps to (0,0).
It's important to note that the physical bounds of the AndroidExternalSurface stay in the correct place and do not move; only the surface's internal content appears to move visually (I don't know how else to describe it).

In the video, you can see the case with a scrollable Column. While I am scrolling through it, the SurfaceView updates correctly. However, when I click on the image to open the fullscreen view, which is another composable added to the layout by an "if" condition, you can notice the original blurred rectangle flickering just before the image opens.

I have updated the issue with these findings. https://issuetracker.google.com/issues/341537869
I hope this information helps with further investigation and fixing the issue. 🙂

surface_view_flicker_add_composable_remove_layout.mp4

[Zach Klippenstein (he/him) [MOD]]

Could be something is initialized to 0,0 and doesn’t get updated until the next frame?

[Sergey Y.]

what do you mean

[Sergey Y.]

If I replace AndroidExternalSurface(SurfaceView) with AndroidEmbeddedExternalSurface(TextureView) everything works fine like any other "canvas" view.

I took a quick look at the AndroidView source code, and I noticed that it always lays out at (0,0). However, this might be okay because it could be positioned relative to the View holder it’s placed in. Hard to say.
https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:compose/ui/ui/src/androidMain/kotlin/androidx/compose/ui/viewinterop/AndroidViewHolder.android.kt;l=261

[Zach Klippenstein (he/him) [MOD]]

I’m just pattern matching that type of behavior. I don’t have a concrete idea what’s actually going wrong in this specific case

[Sergey Y.]

Not sure where to write this message. Maybe you can help me here with androidx.graphics.opengl.GLRenderer
How to initialize it properly with sRGB EGL configuration. Details in the ticket https://issuetracker.google.com/issues/354005994
Thanks in advance.