https://kotlinlang.org logo
Title
c

Colton Idle

05/10/2021, 8:13 PM
This should arguably go into compose, but I'm building a pure desktop compose app (and I've only really done android development), but I'm looking to generate "clouds" as the background of my desktop weather app. I don't really know where to start, but I want my clouds to be generated since desktop will need to support like infinite sizes and so shipping a large video of clouds in the sky won't really work. Do I try to implement this with unity? OpenGL? Something specific in compose? A particle generator? Any pushes in the right direction would be helpful. Image located in thread of the type of cloud I'm looking at generating.
These clouds are almost like smoke that I want scrolling across the screen.
r

romainguy

05/10/2021, 9:33 PM
It depends how fancy you want your clouds to be
You could for instance generate physically based volumetric clouds
Or fake the hell out of it with 2D noise functions
An easy solution would be to just use a various sprites and stack them on top of each other in a somewhat random fashion
It would be easy to implement and cheap
c

Colton Idle

05/10/2021, 10:50 PM
@romainguy this is what my designer gave me. I'm scrubbing the timeline here which shows that the clouds aren't entirely static, but I'd be okay with static at first. Physically based volumetric clouds sounds interesting. The way that the video shows... it's almost like smoke with some of the ends curling, etc. I will play around with 2d noise functions tonight maybe. Is that like an open gl thing or a compose thing? (sorry the most custom stuff I've drawn in my life was a custom background on a button. 😂 so idk where to even start)
r

romainguy

05/10/2021, 10:58 PM
2D noise functions/etc. --> OpenGL
But given this design you can probably do it with a few sprite sheets
c

Colton Idle

05/10/2021, 11:47 PM
Thanks. Will try some openGL! If that fails I'll just try to get a bunch of pngs of clouds and animate them from left to right. 😄
r

romainguy

05/11/2021, 2:03 AM
And rotate/fade them to get those wispy bits
1
c

Colton Idle

05/11/2021, 2:46 PM
@romainguy one last question. If I did this in compose android would I also just try to run OpenGL in an AndroidView, or is there a successor to openGL in compose (could have sworn I read on twitter about openGL going away in compose?)
r

romainguy

05/11/2021, 3:02 PM
Compose has nothing to do with OpenGL
Here's a sample app I wrote that uses GL in Compose
It's based on alpha builds so outdated etc.
c

Colton Idle

05/11/2021, 3:05 PM
Thank you. This helps!
Hey @romainguy, I'm just getting back to this now. My designer basically gave me this to work off of and asked me to recreate it at the bottom of the screen (I tried to get out of it and just use the video that we purchased here: https://elements.envato.com/fog-8SCKJ6V) but it seems difficult to use video with alpha in android. Do you still recommend that I can create this rolling fog effect by just using openGL in my compose app?
r

romainguy

09/11/2021, 4:05 AM
With GL you could just play the video in a surface texture and treat the color as an alpha channel
c

Colton Idle

09/11/2021, 4:08 AM
Really? It seems like just putting the video on top of a blue background for example doesn't have the right effect though. In final cut pro, if I take the video above and put it on a blue background, it comes out all grey looking. I need to add a blend mode (like screen) for it to look good. Unless you're telling me that GL can also do a blend mode?
If so, then all I'm really learning is that I need to learn OpenGL lol. I'm a long time android developer (~5 years) and still haven't really ever had to touch it and so this requirement from design team is killing me.
r

romainguy

09/11/2021, 4:10 AM
So in GL you would do something like this:
vec4 color = texture(my video, UV);
And you have the RGBA color for the video
c

Colton Idle

09/11/2021, 4:10 AM
I was originally trying to play it in my android app using exoplayer, but exoplayer doesn't support alpha. Didn't know that GL could do it: https://github.com/google/ExoPlayer/issues/7789
r

romainguy

09/11/2021, 4:10 AM
Then you can reinterpret that data however you want. For instance:
myOutputColor.a = color.r;
Now you've just used the red channel of the video as the alpha channel for the output. So now you can blend the video properly :)
c

Colton Idle

09/11/2021, 4:11 AM
But cool. Okay. so you think I should be able to just play that video in GL in my android/compose app. Alright. I've got some learning to do. OpenGL is completely foreign to me. Thanks for pointing me in the right direction!
r

romainguy

09/11/2021, 4:11 AM
Or you could do fancier blendings like screen, overlay etc.
c

Colton Idle

09/11/2021, 4:12 AM
Yeah. Screen does work really nicely in final cut pro when I tried it there.
screen blending
no blending
lol
thats final cut pro showing the fog on top of a blue-ish background with screen blending (looks great) vs no blending. And so I'm trying to recreate in android
r

romainguy

09/11/2021, 4:15 AM
Note that if you render into a TextureView you could set the screen mode on that view directly
Maybe you don't need OpenGL at all
Yeah try that
Render the video in a texture view
But lemme check something first
Oooh yeah that'll work
Just call setLayerPaint() on TextureView
And set the screen xfermode on the paint you pass
Done
c

Colton Idle

09/11/2021, 4:18 AM
Oooh. Look at all these things that I'm learning.
Does the movie have to be in any specific format/encoding?
r

romainguy

09/11/2021, 4:20 AM
Nope
Just black and white like your example
And then any video format supported by Android's media codecs will work
When you create the MediaPlayer give it the surface TextureView gives you
And that's all
c

Colton Idle

09/11/2021, 4:22 AM
Cool. Giving it a whirl now. We budgeted about a week of my time to do this, so if I can get this working in one night... then I owe ya... something? lol
r

romainguy

09/11/2021, 4:23 AM
😁
Oh and you may have to setOpaque(false) on TextureView to disable overdraw optimizations I think
👍 1
c

Colton Idle

09/11/2021, 4:35 AM
Okay. So I have my black and white video playing, but the how am I supposed to make the black background transparent?
r

romainguy

09/11/2021, 4:36 AM
You don't. Just set the blend mode to screen or overlay and see what happens
Or add
If none of them work the way you want (but at least one should), I have another idea
c

Colton Idle

09/11/2021, 4:42 AM
gotcha. alright. Gimme a sec... 🤞
no dice.
textureView.surfaceTextureListener = this
        textureView.isOpaque = false
        textureView.backgroundTintBlendMode = BlendMode.SCREEN
        mediaPlayer = MediaPlayer()
        try {
            fileDescriptor = assets.openFd("smoke.mp4")
        }
        catch (e: IOException) {
            e.printStackTrace()
        }
top app bar showing, below is a blue background with the video playing over top, where the black is still visible.
r

romainguy

09/11/2021, 4:51 AM
You need to set the view to not opaque before the surface is created I think
👍 1
And don't set the background tint mode that won't do anything
👍 1
You need to set the xfermode of the Paint you pass to setLayerPaint
1
c

Colton Idle

09/11/2021, 4:55 AM
Wasn't passing anything. So now I need to just learn what I should be passing into setLayerPaint.
Don't see anything about xferMode.
but again. not my forte, so I'm probably missing something basic here.
r

romainguy

09/11/2021, 5:00 AM
Paint().apply { xfermode = PorterDuffXfermode(PorterDuff.Mode.SCREEN) }
Something like that
c

Colton Idle

09/11/2021, 5:01 AM
OHHHHH SHIT
IT'S LIKE KINDA WORKING
r

romainguy

09/11/2021, 5:02 AM
blendMode does the same but it's API 29+
c

Colton Idle

09/11/2021, 5:02 AM
r

romainguy

09/11/2021, 5:02 AM
The javadoc of PorterDuff.Mode shows examples of all the available modes btw
c

Colton Idle

09/11/2021, 5:02 AM
I only said kinda because I need to fix up a few things. mainly... get it looping infinitely.
Had no idea textureView could do all this. lol
Also, the video I got from envato was a
.mov
but that crashes, so it looks like I gotta try to convert the video to a .mp4
r

romainguy

09/11/2021, 5:03 AM
Yeah you'll need an mp4
Or mkv
H265 as mkv will probably be good quality/compression and it works on API 21+
WebM is also a good choice
With VP9 encoding
c

Colton Idle

09/11/2021, 5:07 AM
i think webm with vp9 can actually support transparency too (IIRC). What started this mess was that the design team hear that ios can support alpha videos with hevc. https://developer.apple.com/videos/play/wwdc2019/506
r

romainguy

09/11/2021, 5:08 AM
Yeah VP9 supports alpha but I don't know if it works through the entire pipeline
c

Colton Idle

09/11/2021, 5:17 AM
@romainguy thanks again so much. Could not have done it without you. I may try to write this up in a little blog post. This ended up being simpler than I thought. One last question. Should I be worried about killing battery life if this just would play as the background of my app?
r

romainguy

09/11/2021, 5:21 AM
The video should be decoded in hardware so probably not too bad. It's not ideal but not as bad as playing 3d animations or something I think. You'd have to measure :/
Btw looks like we could also use a ColorMatrixColorFilter to copy a color channel to the alpha channel and create transparency for a video this way
Basically you could green screen it :)
c

Colton Idle

09/11/2021, 5:42 AM
ColorMatrixColorFilter would work with textureView + mediaPlayer? Or is that a different technique in general?
r

romainguy

09/11/2021, 5:47 AM
It would work yes
But it has limitations and would require a carefully crafted video
👍 1
c

Colton Idle

09/11/2021, 5:53 AM
thank you again. porter duff is wild. some of these example inputs and outputs are like magic https://medium.com/mobile-app-development-publication/practical-image-porterduff-mode-usage-in-android-3b4b5d2e8f5f
r

romainguy

09/11/2021, 5:57 AM
Yep they're super useful and the basis of all compositing techniques. Note that what we call Porter-Duff modes in our API contains more than actual Porter-Duff modes (that's why I split them into Porter-Duff and blend modes in the docs). Screen is NOT a Porter-Duff mode for instance :)
👍 1
Also why there's now Paint.setBlendMode() :)
👍 1