https://kotlinlang.org
Join Slack
Does anyone know a workaround to use anti aliasing on scaling images? <https://github.com/JetBrains...
# compose-desktop
s
Does anyone know a workaround to use anti aliasing on scaling images? https://github.com/JetBrains/skiko/issues/896 I tried using the old ImageIO way for creating the thumbnails, but this is way slower and consumes a lot more memory. So I'm looking for a way to still do it with SKIA.
👀 1
✅ 1
a
I’m not really familiar with Skia but I don’t think anti aliasing has anything to do with image scaling, as it’s only involved when drawing lines (that are not perfectly horizontal or vertical) and images are not lines - they are pixels. Isn’t it because you are using the low-quality linear sampling? Try using cubic sampling modes which have higher quality: 
CATMULL_ROM
or 
MITCHELL
.
s
The other sampling modes look a bit better in other regards, but still blocky. It's really the missing anti aliasing.
ImageIO looks far better when setting the anti aliasing rendering hint - for all sampling modes. But it takes longer and consumes more memory.
a
I still believe that anti aliasing is not the problem. Here’s the doc of `SkPaint::setAntiAlias`: > Requests, but does not require, that edge pixels draw opaque or with partial transparency. And here’s the doc of 
Paint.setAntiAlias
in Android (which uses Skia): > AntiAliasing smooths out the edges of what is being drawn, but is has no impact on the interior of the shape. As the doc suggests, anti aliasing only affects edges, and the edges of an image is just a rectengle.
s
Ok, you may be right here. How can I achieve a sampling with anti aliasing then?
Is it not possible?
a
Maybe you can at least provide an example of the result?
s
I want it to look like the version created by GIMP, which obviously uses anti aliasing
GIMP applies the anti aliasing I would expect.
All exported at JPG Quality 90%, so this is not the difference. I compared PNG, too, but the files are big, so I append JPG here.
a
Have you tried 
Image.scalePixels()
?
s
I want to try it, but I don't know how it works unfortunately
Uh, looks like you are right about your guess how anti aliasing in Paint works. https://groups.google.com/g/skia-discuss/c/_ICFWgWXT1w/m/gp0XFM5OAQAJ
a
To be clear, I was talking about how anti aliasing, as a technique, works in general. So the high quality of the image scaled by GIMP is not actually because of anti aliasing. It’s just because of the high-quality scaling algorithm.
s
It's both linear scaling... shouldn't that be the same algorithm / implementation?
a
Yeah so maybe there is some other issue, which is why I asked if you have tried 
Image.scalePixels().
s
Do you know how that function is used? I don't know how I can create a Pixmap.
I shockingly lack a lot of SKIA knowledge 😕
s
On Android, I had an issue while downscaling bitmaps. I originally thought it was an anti-aliasing issue, but it wasn't. I fixed it by downscaling the image by only 50% at a time. It was more time consuming, but my results were more accurate.
🤔 2
When I dug in, this was a common issue, and a common solution. There is a better solution, which is using a better downsampling algorithm, but that's more work, and significantly more nuanced.
s
Or is it that downscaling should always by a magnitude of 2 ratio?
a
I think you can just use something like
Copy code
val pixmap = Pixmap()
image.scalePixels(pixmap, SamplingMode.LINEAR, false)
val scaledImage = Image.makeFromPixmap(pixmap)
s
s
val pixmap = Pixmap()
image.scalePixels(pixmap, SamplingMode.LINEAR, false)
val scaledImage = Image.makeFromPixmap(pixmap)
That would be too easy. 😉 Also there must be a way to specify the target size. 
Failed to Image::makeFromRaster Pixmap(_ptr=0x600002e71170)
a
Ah right 😅 Maybe this?
Copy code
val bitmap = Bitmap()
bitmap.allocN32Pixels(480, 320)
image.scalePixels(bitmap.peekPixels()!!, SamplingMode.LINEAR, false)
val scaledImage = Image.makeFromBitmap(bitmap)
✅ 1
s
Scott is right about that scaling the image multiple times by 50% looks a lot better than one-shot. Here downscaled by factor 4 from a 2400px base image. So might really not be an AA issue.
s
Play around with your algorithm. Try down-scaling 25% at a time and see if that improves things further
s
I assume that 50% is the good number, because making a cluster of 4 pixels to one is easier than doing this with an uneven number.
Did you run those test?
s
This was years ago. I think I did. Once you set up the algorithm, adjusting the max downscaling percent is easy to run the tests
I was only doing this with logos which aren't nearly as detailed as photos
The real solution is to just use a better downscaling algorithm in a case like this. i.e. if you have a browser downscale the image, you'll find that it doesn't have this issue. I'm not sure what that better downscaling algorithm is though. the 
scalePixels
does seem promising though because you can choose the algorithm. Have you gone through all the algorithm options?
s
Yes, all algorithms have the same issue.
s
Do this in a browser though and you'll see it doesn't have this artifacting
s
GIMP does a good job, too
s
so from my perspective we're left with 3 options. 1. Live with the pixelation, 2. Downscale incrementally, which is the inefficient solution, but it's easy to implement. 3. Find a way to use an improved downscaling algorithm that mimics gimp / browser downscaling algorithms. I'm sure you could find an open source version of such an improved algorithm.
👍 1
s
GIMP is open source, yes. So we need to convince SKIA to adopt that.
Ah right 😅 Maybe this?
Yes. That's correct. I has the identical result as the other way. And the same speed. It's less code, so that's good.
s
I had a conversation with ChatGPT. It seems to think using a FilterQuality set to high might solve the problem. I've found it mostly wrong on detailed things like this, but it's worth a shot:
Copy code
fun downscaleImageHighQuality(imageBitmap: ImageBitmap, targetWidth: Int, targetHeight: Int): ImageBitmap {
    val sourceImage = Image.makeFromBitmap(imageBitmap.asDesktopBitmap())
    
    // Create a paint object with high-quality filtering
    val paint = Paint().apply {
        // Set the filter quality to high
        filterQuality = FilterQuality.High
    }
    
    // Perform the resizing with high-quality filtering
    val resizedImage = sourceImage.resize(targetWidth, targetHeight, paint)

    return resizedImage.asImageBitmap()
}
s
That's the same bullshit it told me in the first place. 😄
First of, it's not 
filterQuality
anymore. That's deprecated. It's now 
isAntiAlias
. This has no effect on the image.
Also, there is no resize() method.
s
I wonder if there is an underlying Skia API you could tap into
s
I don't believe that. ChatGPT is just useless on SKIA questions.
It was also not able to tell me how to use 
scalePixels
. I'm so glad we have @Albert Chang with us. 🙏
s
So it looks like Skia has some resampling options. I'd give those a shot.
s
I did that. They suffer all from the same problem.
Mitchell & Catmull look better in some other regards/aspects of the images, but they still develop this blocky look if scaling more than 50%
I would hazard a guess that it does progressive downsampling for any scale factor that is higher than 2.
r
50% is the right number when the filtering algorithm is bilinear filtering which is the default used in Skia. Not sure what's exposed in Skiko/Compose for Desktop but anti-aliasing on images is often used only under rotation to treat edges, filtering is what governs up/downscaling
👀 1
On Android we also added mipmapping to bitmaps which automatically creates the pyramid of 50% downscaled images, check if CfD supports that
s
There is a way to specify it, but in my tests it had no effect.
r
It will only have an effect when you're drawing at a different scale
s
Yes, I downscale 2400px to 480px... Specifying NONE or LINEAR results in the same pixels.
r
Anyway, Michell and Catmull-Rom can be better than bilinear as they use a more complex filtering kernel, but they're not magical and you'll still see artifacts when you downscale a lot without intermediate steps
👍 1
Linear will only sample 4 source pixels for each destination pixel, so it's not surprising you're getting similar results
s
byte identical same results
Oh no!!! That's it.
MipmapMode is ignored on 
drawImageInRect
, but it has an effect on 
scalePixels
Copy code
val bitmap = Bitmap()
bitmap.allocN32Pixels(thumbnailSize.width, thumbnailSize.height)
this.scalePixels(bitmap.peekPixels()!!, FilterMipmap(FilterMode.LINEAR, MipmapMode.LINEAR), false)
return Image.makeFromBitmap(bitmap)
Thank you all for your help! 🙏
r
Yeah so doing it there enables trilinear filtering
Good to know it works when doing a one time scale
s
I wonder why it's broken for drawImageInRect . Until now I was not able to use scalePixels as I did not know what to specify there.
Another post process step should be to sharpen the downscaled images again a bit. They get a bit blurry now.
r
Mitchell/Catmull-Rom/other splines help with that
But that can create a bit of ringing sometimes
s
MITCHELL & CATMULL_ROM don't have the options to specify a MipmapMode .. This is only available for NEAREST & LINEAR
r
Note that mipmapping already helps with sharpness compared to a straight bilinear filter (I have a comparison somewhere)
s
So the strategy is do scale down first using LINEAR with Mipmap and then go the rest with MITCHELL? 🤔
r
Anyway you can run a sharpen pass if you need one
s
How can I do that?
Using only CATMULL from the big image this introduces a lot of blockyness.
r
Look at what image filters are offered in the API, there might be a sharpen one already
If not hopefully there's one that lets you specify you own kernel
s
Yeah, drawInRect offers a Paint
So first scale pixels down and then paining with a filter
This sharps by a lot. 😄
Copy code
surface.canvas.drawImage(
    image = downscaledImage,
    left = 0f,
    top = 0f,
    paint = org.jetbrains.skia.Paint().apply {
        imageFilter = ImageFilter.makeMatrixConvolution(
            kernelW = 3,
            kernelH = 3,
            kernel = floatArrayOf(
                0f, -1f, 0f,
                -1f, 5f, -1f,
                0f, -1f, 0f
            ),
            gain = 1F,
            bias = 0F,
            offsetX = 1,
            offsetY = 1,
            tileMode = FilterTileMode.CLAMP,
            convolveAlpha = false,
            input = null,
            crop = null
        )
    }
)
Copy code
0f, -0.2f, 0f,
-0.2f, 1.8f, -0.2f,
0f, -0.2f, 0f
seems to be a good kernel by trial and error.
Do you have a recommendation for me?
r
With a simple box sharpen like this, anything that works well for your content is good
Anything fancier like unsharp mask is more complicated to implement
s
I don't fully understand how the numbers relate to each other.
Copy code
0f, -0.2f, 0f,
-0.2f, 1.8f, -0.2f,
0f, -0.2f, 0f
was given to me by ChatGPT, but it fails to come up with a kernel that's a little bit softer
r
To soften the sharpen effect you just use values closer to 0 around the center (same value everywhere)
and the center should be 1+(positive sum of the other numbers)
btw i you use something like this:
Copy code
0.0625, 0.125, 0.0625,
0.125, 0.25, 0.125,
0.0625, 0.125, 0.0625
you’ll have a blur
s
Thanks 😊
k
Long time ago in the Java2D land, Romain suggested that the best (quality wise) way to get a thumbnail would be to scale down by exactly 0.5x until you get to the last pass. So if it’s scale down by 0.1x, then it’s 0.5x, another 0.5x (to 0.25x), another 0.5x (to 0.125x), and then 0.8x (to 0.1x)
👍 1
s
I wasn’t aware of this, but it looks like that this multipass halving (MidmapMode) is super common.
Strange that skiko has not enabled it by default.
k
Or maybe Chris Campbell
That one points to https://web.archive.org/web/20070402041751/http://weblogs.java.net/blog/gfx/archive/2006/09/java2d_gradient.html and in the comments section somebody’s cursing at me… Back in 2006. Good times.
r
@Kirill Grouchnikov Chris wrote the blog post after internal discussions with Chet and I on this very topic 🙂 Some things never change I suppose
k
r
Feel weird to see Java2D code again
➕ 1
s
Ok, wow. It really looks like that this knowledge that we need to downscale by half’s until the final step is ancient. So it’s surprising that skiko does not have such a convenience function that automatically detects if it’s a downscale or upscale and applies the correct strategy to it. 🤔 Maybe I should contribute that. 😄
r
It's not "correct" or "incorrect"
It's a more expensive way to improve quality, for some particular definition of quality
👍 1
s
That's now my final code - for now. I still think that GIMP yields better (less blurry) results, but it may be too hard to get there. https://github.com/StefanOltmann/thumbnail-rebuilder/commit/0e6a49cdec3ad3f81623e284e8c04ded05e1345e
r
It’s not about being hard, it’s just a different algorithm with different tradeoffs, esp regarding performance
Not familiar with GIMP but Photoshop lets you choose among several filters when up/downscaling, precisely because it depends on what you are trying to do
s
GIMP has them, too. I compared LINEAR filtering in this case.
But it's no longer a night-and-day difference. GIMP just looks a bit better and I don't know how they achieve that. I may not find it out.
I guess some clever post processing, maybe it's an unsharp you mentioned
With the box sharpening parameters I came closer. Just applying resizing without it's way more blurry than what GIMP has.
Maybe they don't sharp, but have another algorithm that favors quality more. SKIA is indeed super fast and built for speed.
This is how they compare without sharpening.
Already very close.
With a slight sharpening I think it's acceptable. The best I can do right now.
🙌 1
r
that favors quality more.
Again, it's not about quality, it's about what you want the results to be. Sometimes sharpening is not what you want
👀 1
h
in my experience, Libswscale from ffmpeg provides very optimized functions for downscaling an image, in particular the Lanczos algorithm.
👀 1
s
Okay, that's LGPL and maybe not something I can use. But I will scale my sample images with that for a comparison.
I think I will also scale with imagemagick, Affinity Photo, Photoshop and others to see what results I like best to have a reference to what I want to achieve with skiko
h
I've observed performance problems when drawing images created by ImageIO, lag problems in particular. These problems disappeared when I used "skia.Image.makeFromEncoded" instead. I noticed that the difference was that ImageIO creates an image with a color profile, whereas skia does not.
s
Yes, definitely. ImageIO is slower and uses much more memory than skiko. Aside the multiplatform sharing aspect this is a reason why I want to do it with skiko.
It's just that the out-of-the-box results are not too great, so there is some fine-tuning required. For example skiko does not have midmap enabled by default - I wasn't aware what this is before. But I understand why they do it. Without that scaling is much faster.
The performance of SKIA is impressive. On JVM it creates thumbnails even faster than Apple Core Image does on iOS/macOS - at least for the sample images I used.
plus1 1
r
ImageIO drawiny slowly: that's because you need to ensure the decoded image is "compatible", I.e. in a format that the rendering pipeline can handle without conversion. Doing that usually implies copying the decoded image into another one in the right format.
💡 1
Here are several utilities to create such images or convert to compatible images: https://github.com/romainguy/filthy-rich-clients/blob/master/DynamicEffects/Fading/src/GraphicsUtilities.java
@Stefan Oltmann this repository also contains a function somewhere to do the successive downscale by 50% 😁
k
https://www.filthyrichclients.org/ - looks like somebody squatted that domain after it hasn’t been renewed
➕ 1
r
Buy the book 😅
k
Good old times
👍 1
s
Wow, that's quite an impressive resume, Romain Guy. Consider me thoroughly impressed. 💪
The documentation uses the term "compatible image" as if it were self-explanatory, but I find it ambiguous. What exactly does it imply? Compatible with what? I feel this is something I need to look up.
r
Compatible with the rendering context, ie the same pixel format so drawing doesn't need any conversion
k
Does that roughly correspond to Skia’s 
SkColorType
?
r
The Android equivalent is Bitmap.Config
But of course with Java2D the format is basically open ended
s
Without any doubt Affinity Photo Lanczos3 is the best downsampling algorithm I found. That's the result I would want to achieve. It's sharp, but not blocky. SKIA once had a lanczos algorithm, but for some reason it was removed.
I'm wondering if, apart from Skiko, there exists a method to downscale images on the JVM utilizing a Lanczos algorithm with performance and memory consumption comparable to SKIA. A library that lets me do that, but that still has a permissive license like Apache 2 / BSD / MIT
Oh, and I figured out that GIMP once had lanczos and dropped it for something worse. There was some noise I missed back then. I never put so much thought into downscaling images as of today.
k
https://github.com/avaneev/avir might be of interest, but you’d need to wrap the code in some bindings
r
You can find plenty of examples/implementation of Lanczos online (https://github.com/esushinskii/Lanczos for instance)
s
@romainguy That would surely help people understanding how to prepare input data and transform output data. The naked algorithm does not help me. The referenced project comes with no samples. I’m looking for something more ready-to-use. Do you understand that particular project? @Kirill Grouchnikov mentioned AVIR which looks like what I am after. Maybe that’s an interesting project to learn JNI and try to build something. Maybe I do research first if someone already did that.
In the best case SKIA will implement it, because I in that case can directly work with the resized Image. https://issues.skia.org/issues/331406965
Ok, they won’t.
AVIR quality is on-par with Affinity Photo Lanczos3 seperable. Lanczos3 non-separable looks better. But also takes longer of course.
m
Thanks for the thread. For anyone else looking, here is the my final solution to have a good quality downscale based on the above:
Copy code
fun ImageBitmap.scale(width: Int, height: Int): ImageBitmap {
    val image = Image.makeFromBitmap(asSkiaBitmap())
    val scaled = image.scale(width, height)
    return scaled.toComposeImageBitmap()
}

fun Image.scale(width: Int, height: Int): Image {
    val bitmap = Bitmap()
    bitmap.allocN32Pixels(width, height)
    scalePixels(bitmap.peekPixels()!!, FilterMipmap(FilterMode.LINEAR, MipmapMode.LINEAR), false)
    return Image.makeFromBitmap(bitmap)
}
👍 1
479 Views
Open in Slack
PreviousNext
Powered by

kotlinlang

A modern programming language that makes developers happier.

Powered by