Hello everyone, I want to discuss some graphics API limitation and its potential improvement - See T...

# compose

[Sergey Y.]

Hello everyone,
I want to discuss some graphics API limitation and its potential improvement - See Thread for Details 🧵

[Sergey Y.]

In my project, which shares similarities with basic vector editors, I work with vector graphics on a canvas-like layout. This layout, which acts as a canvas with a background grid, contains a collection of vector graphics (termed RenderableGroup). Each RenderableGroup consists of a Path and a Transform, defined by a 3x3 transformation matrix, and is drawn within its own composable function.

As users interact with these graphics—moving, rotating, or resizing them—I compute a temporary transformation matrix that applies only to the visual layer during the action. Once the interaction ends, I use this provisional matrix to calculate the final transformations for the graphic's path and update the UI's final state. This temporary transformation matrix is integrated into the components of Modifier.graphicsLayer {} (as shown in the screenshot).
My challenge arises with graphicsLayer demanding a transformOrigin specification, which dictates the pivot point for transformations. Calculating this pivot point is not straightforward, and the fractional nature of transformOrigin adds complexity.

To address this, I applied the temporary matrix directly to the Canvas used for drawing the path, which works but introduces an additional dependency into the widget hierarchy, diverging from simply modifying the composable graphics layer (as illustrated in another screenshot).

🧵 Could we consider enhancing the Modifier API to include such functionality in the future, or are there technical limitations at present? For comparison, my colleagues working with SwiftUI on iOS have a modifier that performs this task directly, as demonstrated in the accompanying screenshot.

[Sergey Y.]

The bug, incorrect transformOrigin leads to:

[Sergey Y.]

And this is how it works with the canvas using the matrix fix.

[dewildte]

I do not know how to help you but I got to say that this is some pretty cool stuff!

[Sergey Y.]

How can I work around this issue at the Modifier level? 😅

It seems like the transformation matrix should provide enough information. I just need to figure out how to concatenate the specified transformation with the one that's already happening within the composable function.

[Sergey Y.]

Here's the image to provide more context on the type of graphics I'm working with.

[romainguy]

@Nader Jawad

[romainguy]

Hard to tell what’s happening without knowing more about your code, but based on your examples, don’t you want the transform origin to be the default (i.e. the center)?

[Sergey Y.]

Unfortunately, it cannot be centered by default. As you can see in the GIFs, when I grab a corner handle, the transformation is calculated with the opposite handle serving as the anchor point. If I apply this transformation with a different origin, it results in a similar misalignment.

[Nader Jawad]

It seems like you want the transformOrigin to be relative to the handle have you tried doing this?

[Sergey Y.]

When I apply it directly to the canvas, it just works without needing any pivots; the transformation is accurately calculated on the path and fits perfectly

[Nader Jawad]

I don't think you need to apply anything to the canvas.

[Nader Jawad]

Just when the top left handle is held then the transformOrigin should be 0,0

[Nader Jawad]

And in the case of the triangle with the handle at the center the transformOrigin should be 0.5, 0 indicating the middle of the triangle at the top

[Sergey Y.]

It seems like you want the transformOrigin to be relative to the handle have you tried doing this?

Ideally, I wouldn't want to calculate any transformOrigins at all; I'd prefer to simply concatenate it with the drawing matrix.

[Sergey Y.]

And in the case of the triangle with the handle at the center the transformOrigin should be 0.5, 0 indicating the middle of the triangle at the top

I tried. That approach wouldn't work on rotated objects, as their orientation changes the dynamics of the transformation.

[romainguy]

One way we could address this is by having a version of graphicsLayer that takes a matrix directly as the transform

[romainguy]

I don’t think that’s a common feature for most apps but it does help if you somehow have a matrix precomputed by something else

[romainguy]

Extracting scale/rotation/translation properly can be tricky

[Sergey Y.]

One way we could address this is by having a version of graphicsLayer that takes a matrix directly as the transfor

That's exactly what I was looking for, and it's something SwiftUI already offers.

[Sergey Y.]

Extracting scale/rotation/translation properly can be tricky

Can't we just concatenate it as is?

[romainguy]

That’s my point

[romainguy]

if you have a matrix, you have to decompose it, so we can recompose it

[romainguy]

And decomposing it properly is tricky

[Sergey Y.]

Yeah, because of order of operations, origin, etc

[romainguy]

(btw whether SwiftUI has it is irrelevant, we are interested in doing what’s right for Compose and our devs, not what other frameworks are doing)

[Sergey Y.]

Agreed, I just mentioned it because that's where I saw the feature.

[romainguy]

note that we may be limited by what the platform exposes underneath. @Nader Jawad would know if it’s even doable, I forget what RenderNode allows

[Sergey Y.]

I wish I could do it this way:

[romainguy]

isn’t that just a drawWithContent()?

[romainguy]

like, what can’t you do here?

[Sergey Y.]

the drawContent() inside withTransform block

[Sergey Y.]

Or can? 👀

[Sergey Y.]

I haven't tried it yet; will it work as I expect?

[romainguy]

I believe so

[Sergey Y.]

I'll give it a try in a few hours in the morning 😅; it's 3am at my location now, heh.

[Nader Jawad]

The transforms on RenderNode consume translation, rotation and scale parameters that are applied in a specific order. Right now there isn't a way to configure the matrix directly on a RenderNode but you can query the resultant matrix of a RenderNode after those transforms are applied

[mohamed rejeb]

I was working on something similar, and I was drawing everything in a single Canvas composable. Is it better in terms of performance to draw each drawing on a separate composable? I think that I should try this and see the difference.

[Sergey Y.]

I've evaluated this approach as well. In my case, drawing a path is only the tip of the iceberg. It's easier for me to organize and manage everything I need within each composable. I also did some benchmarking; having the drawing inside separate composables rather than on a single canvas doesn't add too much overhead. So, it's fine for me to have it set up this way.

[Sergey Y.]

https://kotlinlang.slack.com/archives/CJLTWPH7S/p1706230459693429?thread_ts=1706227748.463479&cid=CJLTWPH7S

Yay, It did the trick.

[dewildte]

This is so cool!

[Sergey Y.]

lovely

transform_effect.mp4

[dewildte]

I am totally impressed by this on Android!
Doing this natively is so cool.

[Sergey Y.]

I'm just drawing paths on the canvas 😅

[romainguy]

@mohamed rejeb You won't get better performance from using multiple Composables no (but there will be overhead for layout, memory, etc.) You might get better performance if you have very complex elements to draw and many of them to draw and many of them might be off screen. It can however be handled yourself with culling or using render nodes.

[Colton Idle]

oooh. i want to try this app out. let me know when its available

[Sergey Y.]

😅 Ok