I was curious about rendering html into compose di...

# compose-web

[Alex Styl]

I was curious about rendering html into compose directly, so I hacked a way to render <video> from inside of compose without overlaying html.

From the hack I was able to play a video from Compose Resources in 60fps into a Compose Web app (WASM) directly without manipulating the DOM

This allows for overlaying other composables on top of it

Full code and explanation at the HTML Interop issue

EDIT: Should be possible to render any <canvas> directly inside of compose too

compose_video.mov

[Sergey Y.]

How interesting! Out of curiosity, could you try running an animation or clicking on interactive components while the video is playing? I’m just curious whether playing the video this way would impact other elements running alongside it.

[אליהו הדס]

take a look https://kdroidfilter.github.io/ComposeMediaPlayer/sample/

[Alex Styl]

@Sergey Y. not on my computer rn. might try tomorrow cause it's late here. in the demo i am interacting with buttons/clickables that are placed on top of the video. what interactive elements do u mean ?

[Alex Styl]

or rather what kind of scenario do u have in mind

[Sergey Y.]

Nothing concrete, I was just wondering if it might be spamming the main rendering thread or something similar. No worries

[Alex Styl]

worth a more in depth check. for what it's worth the browser tab was at 60 fps when I checked via chrome tools

[Alex Styl]

@אליהו הדס what you shared places a <video> ontop of the compose app, which doesnt work great (ie can't place other composables ontop and a bunch of other issues people mentioned in the HTML interop issue linked).

notice how in the attached video I shared the play/pause buttons are on in front of the video player. the compose app itself does the video rendering instead of placing a <video> on top of the app

[אליהו הדס]

@Alex Styl Check, i can place a composable on top, try to seek the video, you will see a loader on the video

[Alex Styl]

had a look at ur sample again. you are placing the <video> behind the compose app, not in front.

this approach has other type of rendering problems such as not being able to show anything behind the html element (ie say it is semi transparent or animating the z index)

[Alex Styl]

@Sergey Y. seems alright. asked chatgpt for some random animations to try this out. no slow downs whatsoever. the fading is animating the alpha of the composable

EDIT: there is some choppiness on the video on slack, but i think it's slack doing some compression

anims.mov

[Sergey Y.]

thank you

[Skaldebane]

I made a reply in YouTrack, but yeah the problem isn't with the rest of the Compose app (which, as you showed, could continue running fine), but it's that this approach isn't performant. Like, at all. Lots of copying (potentially millions) of pixels, transforming them 3 times between JS Blob** -> Kotlin ByteArray -> Compose Painter.

The moment you don't run this on an M-series MBP, it becomes super obvious. I'm getting 50% CPU usage, with some choppiness on 480p video on a pretty decent laptop (which can play hardware-accelerated 4k without issues). Let alone mobile phones which will choke even worse (and drain the battery).

** I managed to replace the Base64 conversions with directly reading a Blob from the canvas into a bytearray; much faster, but still doesn't solve the underlying issue.

[Skaldebane]

Maybe for stuff like maps, this could be a viable solution. But if a map is anything but an iframe** (e.g. bunch of divs, or canvas), then I think it's better covered by the underlay solution, as it won't impact performance, and we can still route input events from the Compose canvas into it.
(and if it's an iframe, we can't even render it into a canvas).

** from what I've read online, it's not possible to programmatically trigger events on iframes. They can only be interacted with through user-generated events.

[Alex Styl]

That's what PoC are for. it was to showcase how to have compose do the rendering instead of layering html. Performance was irrelevant