I was playing around trying to hook up Tauri’s bac...

# compose-desktop

[אליהו הדס]

I was playing around trying to hook up Tauri’s backend (Tao) with Compose, and it hit me like a truck: the heavyweight part of Compose isn’t the JVM, it’s AWT! Check out these screenshots. The second screenshot is with GraalVM. Originally, the goal was to make it easy to integrate native views into a Compose window. That would significantly improve things like video playback, for example no more need for offscreen rendering, and it would greatly simplify WebView integration without layering issues.

[Sargun Vohra]

I'm also interested in integrating native rendering in a compose app without layering issues

What do you mean by "hook up Tauri’s backend (Tao) with Compose"? I'm familiar with Tauri (electron alternative for apps with Rust core and system webview ui?) so in this case, you're doing rust core with Compose ui?

how does that result in "no more need for offscreen rendering"?

[אליהו הדס]

Tauri uses its own windowing system called Tao, which is completely separate from its WebView. The advantage of Tao is that it provides a cross-platform abstraction, so essentially we wired Compose to Tao once and use it instead of AWT. So, if we set accessibility aside, it’s not a lot of work. Moreover, AWT is what complicates native compilation with GraalVM, and with Tao, that problem disappears as well.

[אליהו הדס]

The current Compose system based on AWT only allows embedding native views through a Swing panel, but doing so breaks layering. You can no longer overlay Compose elements on top of the native view, which makes it practically unusable. This limitation doesn’t exist with Tao.

[אליהו הדס]

I’m really looking for collaborators on this 🙂

[אליהו הדס]

you can check my work on https://github.com/kdroidFilter/Nucleus/tree/wip/tao-experiment

[Sargun Vohra]

ah, so you're rendering Compose inside the Tao window. And your Tao integration somehow provides interop better than SwingPanel

is it at the fidelity of native view interop with Compose on iOS and Android with UIKitView and AndroidView? How does compositing work -- like rendering Compose content with scale/alpha/rotation/clipping over and under the native content, or native content with scale/alpha/rotation/clipping over/under Compose. Does pointer input interop well?

[אליהו הדס]

The goal, of course, is to reach an interop fidelity comparable to Android or iOS, but I haven’t tested that yet. It’s entirely possible though. Right now I’m mostly wiring up the basics: key events, window management, all that foundational stuff. It’s still very early stage.

[אליהו הדס]

My goal is notably to offer a real WebView via WRY (from Tauri as well), and more performant video playback via Compose media player. Otherwise, in my view, interop isn’t necessary; there’s no point in putting a SwiftUI button inside a JVM desktop app that’s meant to be cross-platform by definition.

[אליהו הדס]

Then I observed an incredible startup improvement under the JVM with Tao. I haven’t measured it, but it could potentially make GraalVM compilation completely obsolete.

[אליהו הדס]

Is managing a closed-world compilation mode, and losing CPU throughput performance, really justified for a gain of 70 MB of RAM? That’s a question I need to ask myself.

[Sargun Vohra]

nice!

yeah, with my use case I care most about rendering / compositing interop, not input interop as I'm happy to implement input on the Compose layer as long as Compose captures it all.

my use case is maps, so similar to video playback I have compose components (compass, scale bar, buttons) on top of the native rendered map. And it's important those components composite well visually. But also, the map itself should composite well (for example, any fade transition requires opacity to apply to the native render surface). And OOB that works with iOS and Android TextureView but not Android SurfaceView.

Of course as you've seen, SwingPanel is worse, no Compose overlay at all (well there's an experimental flag but it only works with "lightweight" components -- not native render surfaces). So I'm working now on just doing native rendering to a texture, sampled directly onto a Compose canvas. And it works great on desktop, with a small reflection hack to access some private APIs inside Compose/Skiko that I'm hoping to expose (so they become usable without reflection for kotlin native and wasm)

[Sargun Vohra]

once I'm done with my current project (wrapping maplibre native with low level C bindings, to then provide low level unified `expect`/`actual` KMP bindings across targets and streamline maplibre-compose's internals), perhaps I'll try prototyping maplibre-compose on your Compose+Tao

[אליהו הדס]

I plan to simply expose a native view API, it should be very straightforward.

[אליהו הדס]

There’s also something else that would be interesting, but which personally interests me much less: wiring Tao with native Kotlin would be very simple, but I find it pretty pointless. You’d completely lose the Java ecosystem and end up with rather mediocre performance compared to GraalVM Native Image, all just to save around 20 MB in bundle size.

[Michael Paus]

That would also be really useful for your PDF viewer, wouldn't it?

[אליהו הדס]

Not really, unless you just want to gain 50MB of RAM.

[אליהו הדס]

My PDF reader doesn't use the system's native view; it decodes it via PDFium.

[Michael Paus]

I know but I thought that this is basically the same scenario as with @Sargun Vohra’s MapLibre.