https://kotlinlang.org logo
#compose-desktop
Title
# compose-desktop
s

Stefan Oltmann

03/22/2024, 10:51 AM
Does anyone know a workaround to use anti aliasing on scaling images? https://github.com/JetBrains/skiko/issues/896 I tried using the old ImageIO way for creating the thumbnails, but this is way slower and consumes a lot more memory. So I'm looking for a way to still do it with SKIA.
👀 1
1
a

Albert Chang

03/22/2024, 12:17 PM
I’m not really familiar with Skia but I don’t think anti aliasing has anything to do with image scaling, as it’s only involved when drawing lines (that are not perfectly horizontal or vertical) and images are not lines - they are pixels. Isn’t it because you are using the low-quality linear sampling? Try using cubic sampling modes which have higher quality:
CATMULL_ROM
or
MITCHELL
.
s

Stefan Oltmann

03/22/2024, 12:28 PM
The other sampling modes look a bit better in other regards, but still blocky. It's really the missing anti aliasing.
ImageIO looks far better when setting the anti aliasing rendering hint - for all sampling modes. But it takes longer and consumes more memory.
a

Albert Chang

03/22/2024, 12:57 PM
I still believe that anti aliasing is not the problem. Here’s the doc of `SkPaint::setAntiAlias`: > Requests, but does not require, that edge pixels draw opaque or with partial transparency. And here’s the doc of
Paint.setAntiAlias
in Android (which uses Skia): > AntiAliasing smooths out the edges of what is being drawn, but is has no impact on the interior of the shape. As the doc suggests, anti aliasing only affects edges, and the edges of an image is just a rectengle.
s

Stefan Oltmann

03/22/2024, 12:59 PM
Ok, you may be right here. How can I achieve a sampling with anti aliasing then?
Is it not possible?
a

Albert Chang

03/22/2024, 1:08 PM
Maybe you can at least provide an example of the result?
s

Stefan Oltmann

03/22/2024, 1:18 PM
I want it to look like the version created by GIMP, which obviously uses anti aliasing
GIMP applies the anti aliasing I would expect.
All exported at JPG Quality 90%, so this is not the difference. I compared PNG, too, but the files are big, so I append JPG here.
a

Albert Chang

03/22/2024, 1:32 PM
Have you tried
Image.scalePixels()
?
s

Stefan Oltmann

03/22/2024, 1:32 PM
I want to try it, but I don't know how it works unfortunately
Uh, looks like you are right about your guess how anti aliasing in Paint works. https://groups.google.com/g/skia-discuss/c/_ICFWgWXT1w/m/gp0XFM5OAQAJ
a

Albert Chang

03/22/2024, 1:39 PM
To be clear, I was talking about how anti aliasing, as a technique, works in general. So the high quality of the image scaled by GIMP is not actually because of anti aliasing. It’s just because of the high-quality scaling algorithm.
s

Stefan Oltmann

03/22/2024, 1:40 PM
It's both linear scaling... shouldn't that be the same algorithm / implementation?
a

Albert Chang

03/22/2024, 1:42 PM
Yeah so maybe there is some other issue, which is why I asked if you have tried
Image.scalePixels().
s

Stefan Oltmann

03/22/2024, 1:53 PM
Do you know how that function is used? I don't know how I can create a Pixmap.
I shockingly lack a lot of SKIA knowledge 😕
s

spierce7

03/22/2024, 1:56 PM
On Android, I had an issue while downscaling bitmaps. I originally thought it was an anti-aliasing issue, but it wasn't. I fixed it by downscaling the image by only 50% at a time. It was more time consuming, but my results were more accurate.
🤔 2
When I dug in, this was a common issue, and a common solution. There is a better solution, which is using a better downsampling algorithm, but that's more work, and significantly more nuanced.
s

Stefan Oltmann

03/22/2024, 1:59 PM
Or is it that downscaling should always by a magnitude of 2 ratio?
a

Albert Chang

03/22/2024, 1:59 PM
I think you can just use something like
Copy code
val pixmap = Pixmap()
image.scalePixels(pixmap, SamplingMode.LINEAR, false)
val scaledImage = Image.makeFromPixmap(pixmap)
s

Stefan Oltmann

03/22/2024, 2:11 PM
val pixmap = Pixmap()
image.scalePixels(pixmap, SamplingMode.LINEAR, false)
val scaledImage = Image.makeFromPixmap(pixmap)
That would be too easy. 😉 Also there must be a way to specify the target size.
Failed to Image::makeFromRaster Pixmap(_ptr=0x600002e71170)
a

Albert Chang

03/22/2024, 2:22 PM
Ah right 😅 Maybe this?
Copy code
val bitmap = Bitmap()
bitmap.allocN32Pixels(480, 320)
image.scalePixels(bitmap.peekPixels()!!, SamplingMode.LINEAR, false)
val scaledImage = Image.makeFromBitmap(bitmap)
1
s

Stefan Oltmann

03/22/2024, 2:32 PM
Scott is right about that scaling the image multiple times by 50% looks a lot better than one-shot. Here downscaled by factor 4 from a 2400px base image. So might really not be an AA issue.
s

spierce7

03/22/2024, 2:33 PM
Play around with your algorithm. Try down-scaling 25% at a time and see if that improves things further
s

Stefan Oltmann

03/22/2024, 2:34 PM
I assume that 50% is the good number, because making a cluster of 4 pixels to one is easier than doing this with an uneven number.
Did you run those test?
s

spierce7

03/22/2024, 2:35 PM
This was years ago. I think I did. Once you set up the algorithm, adjusting the max downscaling percent is easy to run the tests
I was only doing this with logos which aren't nearly as detailed as photos
The real solution is to just use a better downscaling algorithm in a case like this. i.e. if you have a browser downscale the image, you'll find that it doesn't have this issue. I'm not sure what that better downscaling algorithm is though. the
scalePixels
does seem promising though because you can choose the algorithm. Have you gone through all the algorithm options?
s

Stefan Oltmann

03/22/2024, 2:37 PM
Yes, all algorithms have the same issue.
s

spierce7

03/22/2024, 2:38 PM
Do this in a browser though and you'll see it doesn't have this artifacting
s

Stefan Oltmann

03/22/2024, 2:39 PM
GIMP does a good job, too
s

spierce7

03/22/2024, 2:41 PM
so from my perspective we're left with 3 options. 1. Live with the pixelation, 2. Downscale incrementally, which is the inefficient solution, but it's easy to implement. 3. Find a way to use an improved downscaling algorithm that mimics gimp / browser downscaling algorithms. I'm sure you could find an open source version of such an improved algorithm.
👍 1
s

Stefan Oltmann

03/22/2024, 2:43 PM
GIMP is open source, yes. So we need to convince SKIA to adopt that.
Ah right 😅 Maybe this?
Yes. That's correct. I has the identical result as the other way. And the same speed. It's less code, so that's good.
s

spierce7

03/22/2024, 2:49 PM
I had a conversation with ChatGPT. It seems to think using a FilterQuality set to high might solve the problem. I've found it mostly wrong on detailed things like this, but it's worth a shot:
Copy code
fun downscaleImageHighQuality(imageBitmap: ImageBitmap, targetWidth: Int, targetHeight: Int): ImageBitmap {
    val sourceImage = Image.makeFromBitmap(imageBitmap.asDesktopBitmap())
    
    // Create a paint object with high-quality filtering
    val paint = Paint().apply {
        // Set the filter quality to high
        filterQuality = FilterQuality.High
    }
    
    // Perform the resizing with high-quality filtering
    val resizedImage = sourceImage.resize(targetWidth, targetHeight, paint)

    return resizedImage.asImageBitmap()
}
s

Stefan Oltmann

03/22/2024, 2:50 PM
That's the same bullshit it told me in the first place. 😄
First of, it's not
filterQuality
anymore. That's deprecated. It's now
isAntiAlias
. This has no effect on the image.
Also, there is no resize() method.
s

spierce7

03/22/2024, 2:51 PM
I wonder if there is an underlying Skia API you could tap into
s

Stefan Oltmann

03/22/2024, 2:52 PM
I don't believe that. ChatGPT is just useless on SKIA questions.
It was also not able to tell me how to use
scalePixels
. I'm so glad we have @Albert Chang with us. 🙏
s

spierce7

03/22/2024, 2:54 PM
So it looks like Skia has some resampling options. I'd give those a shot.
s

Stefan Oltmann

03/22/2024, 2:55 PM
I did that. They suffer all from the same problem.
Mitchell & Catmull look better in some other regards/aspects of the images, but they still develop this blocky look if scaling more than 50%
I would hazard a guess that it does progressive downsampling for any scale factor that is higher than 2.
r

romainguy

03/22/2024, 3:06 PM
50% is the right number when the filtering algorithm is bilinear filtering which is the default used in Skia. Not sure what's exposed in Skiko/Compose for Desktop but anti-aliasing on images is often used only under rotation to treat edges, filtering is what governs up/downscaling
👀 1
On Android we also added mipmapping to bitmaps which automatically creates the pyramid of 50% downscaled images, check if CfD supports that
s

Stefan Oltmann

03/22/2024, 3:08 PM
There is a way to specify it, but in my tests it had no effect.
r

romainguy

03/22/2024, 3:09 PM
It will only have an effect when you're drawing at a different scale
s

Stefan Oltmann

03/22/2024, 3:10 PM
Yes, I downscale 2400px to 480px... Specifying NONE or LINEAR results in the same pixels.
r

romainguy

03/22/2024, 3:10 PM
Anyway, Michell and Catmull-Rom can be better than bilinear as they use a more complex filtering kernel, but they're not magical and you'll still see artifacts when you downscale a lot without intermediate steps
👍 1
Linear will only sample 4 source pixels for each destination pixel, so it's not surprising you're getting similar results
s

Stefan Oltmann

03/22/2024, 3:11 PM
byte identical same results
Oh no!!! That's it.
MipmapMode is ignored on
drawImageInRect
, but it has an effect on
scalePixels
Copy code
val bitmap = Bitmap()
bitmap.allocN32Pixels(thumbnailSize.width, thumbnailSize.height)
this.scalePixels(bitmap.peekPixels()!!, FilterMipmap(FilterMode.LINEAR, MipmapMode.LINEAR), false)
return Image.makeFromBitmap(bitmap)
Thank you all for your help! 🙏
r

romainguy

03/22/2024, 3:31 PM
Yeah so doing it there enables trilinear filtering
Good to know it works when doing a one time scale
s

Stefan Oltmann

03/22/2024, 3:32 PM
I wonder why it's broken for drawImageInRect . Until now I was not able to use scalePixels as I did not know what to specify there.
Another post process step should be to sharpen the downscaled images again a bit. They get a bit blurry now.
r

romainguy

03/22/2024, 3:36 PM
Mitchell/Catmull-Rom/other splines help with that
But that can create a bit of ringing sometimes
s

Stefan Oltmann

03/22/2024, 3:36 PM
MITCHELL & CATMULL_ROM don't have the options to specify a MipmapMode .. This is only available for NEAREST & LINEAR
r

romainguy

03/22/2024, 3:37 PM
Note that mipmapping already helps with sharpness compared to a straight bilinear filter (I have a comparison somewhere)
s

Stefan Oltmann

03/22/2024, 3:37 PM
So the strategy is do scale down first using LINEAR with Mipmap and then go the rest with MITCHELL? 🤔
r

romainguy

03/22/2024, 3:37 PM
Anyway you can run a sharpen pass if you need one
s

Stefan Oltmann

03/22/2024, 3:37 PM
How can I do that?
Using only CATMULL from the big image this introduces a lot of blockyness.
r

romainguy

03/22/2024, 3:38 PM
Look at what image filters are offered in the API, there might be a sharpen one already
If not hopefully there's one that lets you specify you own kernel
s

Stefan Oltmann

03/22/2024, 3:39 PM
Yeah, drawInRect offers a Paint
So first scale pixels down and then paining with a filter
This sharps by a lot. 😄
Copy code
surface.canvas.drawImage(
    image = downscaledImage,
    left = 0f,
    top = 0f,
    paint = org.jetbrains.skia.Paint().apply {
        imageFilter = ImageFilter.makeMatrixConvolution(
            kernelW = 3,
            kernelH = 3,
            kernel = floatArrayOf(
                0f, -1f, 0f,
                -1f, 5f, -1f,
                0f, -1f, 0f
            ),
            gain = 1F,
            bias = 0F,
            offsetX = 1,
            offsetY = 1,
            tileMode = FilterTileMode.CLAMP,
            convolveAlpha = false,
            input = null,
            crop = null
        )
    }
)
Copy code
0f, -0.2f, 0f,
-0.2f, 1.8f, -0.2f,
0f, -0.2f, 0f
seems to be a good kernel by trial and error.
Do you have a recommendation for me?
r

romainguy

03/22/2024, 4:01 PM
With a simple box sharpen like this, anything that works well for your content is good
Anything fancier like unsharp mask is more complicated to implement
s

Stefan Oltmann

03/22/2024, 4:04 PM
I don't fully understand how the numbers relate to each other.
Copy code
0f, -0.2f, 0f,
-0.2f, 1.8f, -0.2f,
0f, -0.2f, 0f
was given to me by ChatGPT, but it fails to come up with a kernel that's a little bit softer
r

romainguy

03/22/2024, 4:17 PM
To soften the sharpen effect you just use values closer to 0 around the center (same value everywhere)
and the center should be 1+(positive sum of the other numbers)
btw i you use something like this:
Copy code
0.0625, 0.125, 0.0625,
0.125, 0.25, 0.125,
0.0625, 0.125, 0.0625
you’ll have a blur
s

Stefan Oltmann

03/22/2024, 5:07 PM
Thanks 😊
k

Kirill Grouchnikov

03/22/2024, 8:23 PM
Long time ago in the Java2D land, Romain suggested that the best (quality wise) way to get a thumbnail would be to scale down by exactly 0.5x until you get to the last pass. So if it’s scale down by 0.1x, then it’s 0.5x, another 0.5x (to 0.25x), another 0.5x (to 0.125x), and then 0.8x (to 0.1x)
👍 1
s

Stefan Oltmann

03/22/2024, 8:24 PM
I wasn’t aware of this, but it looks like that this multipass halving (MidmapMode) is super common.
Strange that skiko has not enabled it by default.
k

Kirill Grouchnikov

03/22/2024, 8:25 PM
Or maybe Chris Campbell
That one points to https://web.archive.org/web/20070402041751/http://weblogs.java.net/blog/gfx/archive/2006/09/java2d_gradient.html and in the comments section somebody’s cursing at me… Back in 2006. Good times.
r

romainguy

03/22/2024, 8:31 PM
@Kirill Grouchnikov Chris wrote the blog post after internal discussions with Chet and I on this very topic 🙂 Some things never change I suppose
r

romainguy

03/22/2024, 8:41 PM
Feel weird to see Java2D code again
1
s

Stefan Oltmann

03/22/2024, 9:04 PM
Ok, wow. It really looks like that this knowledge that we need to downscale by half’s until the final step is ancient. So it’s surprising that skiko does not have such a convenience function that automatically detects if it’s a downscale or upscale and applies the correct strategy to it. 🤔 Maybe I should contribute that. 😄
r

romainguy

03/22/2024, 9:12 PM
It's not "correct" or "incorrect"
It's a more expensive way to improve quality, for some particular definition of quality
👍 1
s

Stefan Oltmann

03/22/2024, 10:22 PM
That's now my final code - for now. I still think that GIMP yields better (less blurry) results, but it may be too hard to get there. https://github.com/StefanOltmann/thumbnail-rebuilder/commit/0e6a49cdec3ad3f81623e284e8c04ded05e1345e
r

romainguy

03/22/2024, 10:25 PM
It’s not about being hard, it’s just a different algorithm with different tradeoffs, esp regarding performance
Not familiar with GIMP but Photoshop lets you choose among several filters when up/downscaling, precisely because it depends on what you are trying to do
s

Stefan Oltmann

03/22/2024, 10:28 PM
GIMP has them, too. I compared LINEAR filtering in this case.
But it's no longer a night-and-day difference. GIMP just looks a bit better and I don't know how they achieve that. I may not find it out.
I guess some clever post processing, maybe it's an unsharp you mentioned
With the box sharpening parameters I came closer. Just applying resizing without it's way more blurry than what GIMP has.
Maybe they don't sharp, but have another algorithm that favors quality more. SKIA is indeed super fast and built for speed.
This is how they compare without sharpening.
Already very close.
With a slight sharpening I think it's acceptable. The best I can do right now.
🙌 1
r

romainguy

03/23/2024, 8:04 PM
that favors quality more.
Again, it's not about quality, it's about what you want the results to be. Sometimes sharpening is not what you want
👀 1
h

Henri Gourvest

03/26/2024, 12:15 PM
in my experience, Libswscale from ffmpeg provides very optimized functions for downscaling an image, in particular the Lanczos algorithm.
👀 1
s

Stefan Oltmann

03/26/2024, 12:17 PM
Okay, that's LGPL and maybe not something I can use. But I will scale my sample images with that for a comparison.
I think I will also scale with imagemagick, Affinity Photo, Photoshop and others to see what results I like best to have a reference to what I want to achieve with skiko
h

Henri Gourvest

03/26/2024, 12:22 PM
I've observed performance problems when drawing images created by ImageIO, lag problems in particular. These problems disappeared when I used "skia.Image.makeFromEncoded" instead. I noticed that the difference was that ImageIO creates an image with a color profile, whereas skia does not.
s

Stefan Oltmann

03/26/2024, 12:38 PM
Yes, definitely. ImageIO is slower and uses much more memory than skiko. Aside the multiplatform sharing aspect this is a reason why I want to do it with skiko.
It's just that the out-of-the-box results are not too great, so there is some fine-tuning required. For example skiko does not have midmap enabled by default - I wasn't aware what this is before. But I understand why they do it. Without that scaling is much faster.
The performance of SKIA is impressive. On JVM it creates thumbnails even faster than Apple Core Image does on iOS/macOS - at least for the sample images I used.
plus1 1
r

romainguy

03/26/2024, 3:20 PM
ImageIO drawiny slowly: that's because you need to ensure the decoded image is "compatible", I.e. in a format that the rendering pipeline can handle without conversion. Doing that usually implies copying the decoded image into another one in the right format.
💡 1
Here are several utilities to create such images or convert to compatible images: https://github.com/romainguy/filthy-rich-clients/blob/master/DynamicEffects/Fading/src/GraphicsUtilities.java
@Stefan Oltmann this repository also contains a function somewhere to do the successive downscale by 50% 😁
k

Kirill Grouchnikov

03/26/2024, 4:29 PM
https://www.filthyrichclients.org/ - looks like somebody squatted that domain after it hasn’t been renewed
1
r

romainguy

03/26/2024, 4:31 PM
Buy the book 😅
k

Kirill Grouchnikov

03/26/2024, 4:55 PM
Good old times
👍 1
s

Stefan Oltmann

03/26/2024, 7:11 PM
Wow, that's quite an impressive resume, Romain Guy. Consider me thoroughly impressed. 💪
The documentation uses the term "compatible image" as if it were self-explanatory, but I find it ambiguous. What exactly does it imply? Compatible with what? I feel this is something I need to look up.
r

romainguy

03/26/2024, 8:21 PM
Compatible with the rendering context, ie the same pixel format so drawing doesn't need any conversion
k

Kirill Grouchnikov

03/26/2024, 10:04 PM
Does that roughly correspond to Skia’s
SkColorType
?
r

romainguy

03/26/2024, 10:34 PM
The Android equivalent is Bitmap.Config
But of course with Java2D the format is basically open ended
s

Stefan Oltmann

03/27/2024, 1:18 PM
Without any doubt Affinity Photo Lanczos3 is the best downsampling algorithm I found. That's the result I would want to achieve. It's sharp, but not blocky. SKIA once had a lanczos algorithm, but for some reason it was removed.
I'm wondering if, apart from Skiko, there exists a method to downscale images on the JVM utilizing a Lanczos algorithm with performance and memory consumption comparable to SKIA. A library that lets me do that, but that still has a permissive license like Apache 2 / BSD / MIT
Oh, and I figured out that GIMP once had lanczos and dropped it for something worse. There was some noise I missed back then. I never put so much thought into downscaling images as of today.
k

Kirill Grouchnikov

03/27/2024, 3:15 PM
https://github.com/avaneev/avir might be of interest, but you’d need to wrap the code in some bindings
r

romainguy

03/27/2024, 3:18 PM
You can find plenty of examples/implementation of Lanczos online (https://github.com/esushinskii/Lanczos for instance)
s

Stefan Oltmann

03/27/2024, 4:42 PM
@romainguy That would surely help people understanding how to prepare input data and transform output data. The naked algorithm does not help me. The referenced project comes with no samples. I’m looking for something more ready-to-use. Do you understand that particular project? @Kirill Grouchnikov mentioned AVIR which looks like what I am after. Maybe that’s an interesting project to learn JNI and try to build something. Maybe I do research first if someone already did that.
In the best case SKIA will implement it, because I in that case can directly work with the resized Image. https://issues.skia.org/issues/331406965
Ok, they won’t.
AVIR quality is on-par with Affinity Photo Lanczos3 seperable. Lanczos3 non-separable looks better. But also takes longer of course.
4 Views