https://kotlinlang.org logo
Title
a

Anivie

04/19/2023, 10:41 AM
Hi everyone, I want to use a media resource in my desktop application. However, I have noticed that there seems to be no MediaView component in Compose at the moment. As a result, I am forced to search for a component similar to Android's SurfaceView or JavaFx's PixelBuffer in order to use FFmpeg in C for decoding and render the decoded data into my Compose application. Unfortunately, it appears that Compose does not currently provide such a component. May I inquire as to when a similar component might be added?
s

Stylianos Gakis

04/19/2023, 11:15 AM
A lot of words I am not familiar with in this message, but reminded me of this thread here https://kotlinlang.slack.com/archives/CJLTWPH7S/p1677599380551119. Just linking it in case it’s useful to you.
m

Michael Paus

04/19/2023, 11:43 AM
Yes, it would be great to have such an API in Compose. I once wrote the code to connect the VLC player to JavaFX via such a PixelBuffer.
a

Anivie

04/19/2023, 12:08 PM
Well, thanks @Stylianos Gakis. Indeed, this is a similar issue. However, he is working on an Android device, so he can use the API provided by Android to resolve the issue. In my case, I am working on a desktop device and do not wish to use Swing that provide the smae API.
And, I share the same requirement with him: Compose should provide us a native rendering interface to render data that we decoded by other way.
l

Landry Norris

04/19/2023, 1:58 PM
I (unfortunately) can attest that it's 100% possible to use ffmpeg to decode the frames and render to compose. This was at my last job, so I can't share any code.
I was able to create a Compose Image on Android and iOS, then render that to an image. I'd imagine desktop will be similar to iOS.
FFmpeg is a very difficult library to learn, with a lot of codec-specific caveats. I'd highly recommend using Swing interop unless you have a very specific reason to write your own video player, like we did.
a

Anivie

04/19/2023, 3:37 PM
Could you please inform me which component you are using?
l

Landry Norris

04/19/2023, 3:40 PM
I don't remember what method I used to create the image. It was definitely specific to skiko.
a

Anivie

04/19/2023, 3:41 PM
If you are using a canvas (or a similar component), it is possible, but unlike the component I mentioned in the example, others can render images directly from the decoded data, whereas canvas requires manual drawing in the JVM. It's worth noting that if you only need to display a single image, then canvas is acceptable, but if I want to display consecutive frames for playing a video, then the performance of canvas is clearly inadequate.
Are you utilizing an API present in Skia? If so, If so, I may not know how it was implemented because I haven't studied Skia before
l

Landry Norris

04/19/2023, 3:43 PM
Have you tested that? The performance was great on iOS when I tested it. On Android, the canvas drawing was quite fast (the main limiter was ffmpeg). I could render at about 60fps using the canvas.
a

Anivie

04/19/2023, 3:48 PM
Did you mean
androidx.compose.ui.graphics.Canvas
?
l

Landry Norris

04/19/2023, 3:48 PM
Yes
I remember having to make some optimizations to get that performance (mainly regarding param stability and remembering), but it's been a minute.
a

Anivie

04/19/2023, 3:52 PM
Ok, thanks for the answer, I'll try using Canvas
But I don't think using Canvas for external drawing is the best for either performance or portability, so I'd still like compose to provide an API that allows us to do this more efficiently.
l

Landry Norris

04/19/2023, 3:56 PM
I agree. It would be great to be able to have a SKCanvas as a composable. That's what they should be backed with in the end anyways.
m

Michael Paus

04/19/2023, 4:20 PM
You know that you can get access to the native canvas via Canvas.nativeCanvas? Have you checked that API? It has a lot of fancy methods which you don’t find in the Compose Canvas.
a

Anivie

04/19/2023, 4:30 PM
OK, that's a good idea. Thank you for your answer
l

Landry Norris

04/19/2023, 4:40 PM
I remember trying nativeCanvas, but it didn't have what I needed at the time. It may be different now.
a

Anivie

04/19/2023, 4:42 PM
OK,thanks