<Kompress> 2.0.0 for Kotlin 2.4.0 is out now! High...

# feed

[Alexander Hinze]

Kompress 2.0.0 for Kotlin 2.4.0 is out now!
Highlights for this release:
• Pure Kotlin implementation of core (de)compression APIs like Deflater/Inflater
• Full GZIP support
• Experimental ZIP support (only archiving at the moment)
• Improved test coverage, including round-trip tests using Java's standard APIs.
https://github.com/karmakrafts/Kompress/releases

[altavir]

I see that you have benchmark comparison with JRE implementation. Do you have results published anywhere?

[Alexander Hinze]

As the new implementation is still fairly new i didn't publish complete results yet, since the first results were inaccurate, and i don't want to publish wrong results. However, i can run some current numbers for you until the next update 🙂

[altavir]

It would be interesting. Multiplatform ZIP is important feature. Even if performance is worse. But I wonder if it is...

[Alexander Hinze]

I can tell you i am fairly confident the speed on JVM is very much acceptable, if not better in some cases 🙂 JS is horrible tho, but you can use WASM from JS no problem. Benchmarks are running right now.
I am disabling the experimental jdk.incubator.vector support for these tests, as performance still varies strongly with it enabled.

[Alexander Hinze]

Lemme make a quick graph to make these numbers digestible

[Alexander Hinze]

On my old Threadripper 1950X dev machine, these are the current results. I basicly beat the JVM in every setting. Of course this is highly synthetic and assumes perfectly compressible data. I will throw together some real world benchmarks too using actual files for the next update 🙂

[Alexander Hinze]

Decompression seems to be on a libdeflate level performance wise at this point

[Alexander Hinze]

I am pretty sure my implementation uses more memory tho compared to java.util.zip as that's just a JNI layer on top of zlib

[Alexander Hinze]

And i am also playing around with SIMD optimizations for the decompression, but we'll see how that goes 😄

[altavir]

An increase for unzip is significant. Also default vectorization is rather good. It is not that deterministic, but in my experience, there are no problesm with it.

[Alexander Hinze]

Oh i know the default is pretty good, but i could already get a couple hundred MiB/s extra for compression with my experimental implementation. It's just an experiment right now, but i definitly see potential to make it even faster using well written vector code.

[Stefan Oltmann]

I'm looking to replace Pako (used through Kotlin/JS bindings). Do you happen to have numbers on that?

[Alexander Hinze]

So you want to bind the WASM version of the lib to replace Pako, am i getting that right? ^^ Because i strongly advise against the JS version hahahaha. It is really slow at the moment.

[Stefan Oltmann]

I was more thinking of getting rid of JS bindings and use the lib from commonMain like any other KMP lib.

[Alexander Hinze]

Well, the JS version is pretty slow i fear, but WASM should be very solid.

[Stefan Oltmann]

https://github.com/StefanOltmann/kim/blob/e073af0faa8a29173717b8ce5693670e27a9e217/src/jsMain/kotlin/de/stefan_oltmann/kim/common/ZLib.js.kt#L22

[Alexander Hinze]

I'll kick off a multiplatform benchmark, gimme a moment

[Stefan Oltmann]

Also https://github.com/StefanOltmann/kim/blob/e073af0faa8a29173717b8ce5693670e27a9e217/src/wasmMain/kotlin/de/stefan_oltmann/kim/common/ZLib.wasm.kt

[Alexander Hinze]

I see, yeah v1 of the lib did it like that too using fflate hahaha

[Alexander Hinze]

problem with every single deflate lib for JS/WASM out there is, that they don't allow partial decompression, or at least they don't expose it in their APIs, so i don't know how many bytes the inflater actually read and where it stopped. Without that feature its impossible to implement archive container formats

[Alexander Hinze]

at least in a streaming manner, which is what you want if possible

[Alexander Hinze]

Alright, running a platform comparison benchmark right now ^^ Gonna take a moment

[Stefan Oltmann]

Is fflate better than Pako?
I always read that Pako was the recommended lib to do it.

[Alexander Hinze]

I honestly don't really know, fflate was just the first thing that i found and that worked at the time to be honest.

[Stefan Oltmann]

A good reason to compare fflate, Pako & yours

[Alexander Hinze]

Okay so,
on JS, compression manages about 60-65MiB/s and decompression about 150MiB/s on default levels.
on WASM, compression manages about 77MiB/s and decompression about 500MiB/s on default levels.
Not as bad as i thought to be honest.

[Alexander Hinze]

so the bottleneck on JS is definitly not JS, but my compression implementation, which is still a lot slower than my decompression.

[Alexander Hinze]

but i focused on the latter first, it shows

[Stefan Oltmann]

WASM is really much faster than JS only. For my libs numbers on WASM are also much better.

[Alexander Hinze]

Yep, but it makes sense. The compression processes bitstreams on kotlin level, and on JS you don't have any fancy intrinsics like JVM or native for those things. You can just hope the Kotlin/JS compiler produces the next closest thing to ASM.JS code hahaha.

[Alexander Hinze]

Actually now that i think of it.. I could probably use js() inline code in my bitstream lib to optimize using handwritten ASM.JS 😁

[Stefan Oltmann]

On WASM I‘d like to see how yours compares to Pako. That tells me what I gain.

[Alexander Hinze]

I can add some comparison benchmarks at some point if you like ^^

[altavir]

In my experience, Js could produce very good performance if one uses webgl buffers instead of JS types.

[Stefan Oltmann]

Would be helpful. If I manage to migrate to your lib I'll report my findings.

[Alexander Hinze]

@altavir That is absolutely true! But i try to use as much of kotlinx.io.core that i can, their Buffers are quite nice to work with and build APIs around with Sinks and Sources. Maybe i can use this as an internal optimization only, like the vector stuff on JVM.

[Alexander Hinze]

My bitstream library is also built on kotlinx.io, so everything goes through these buffers. Definitly lots of potential for optimizations

[Stefan Oltmann]

Same for me. In the next version of Kim I want to drop my custom ByteReader and ByteWriter. Should be replaced with the official types (Source and Sink).

[Alexander Hinze]

I am about to publish an async layer for kotlinx.io, maybe you wanna have a look at that once it's out? ^^ I think it will make kotlin web stuff way less tedious.

[Alexander Hinze]

It's based on the official async proposal and implements it on all platforms + extra utils/helpers

[Stefan Oltmann]

Not sure for what I would use it to be honest.
My libs do synced IO.

[Alexander Hinze]

Oh okay fair ^^ It was more like a plug for my project if you do want to have async IO at some point 😁

[altavir]

async layer for kotlinx-io is ktor-io. It has a lot of problems and you should use it only if you are sure you need it. Let's move this discussion to #CKCAAV8VB

[Alexander Hinze]

@altavir That is not what i meant. I meant i wrote a proper async layer that's built on top of kotlinx.io which has nothing to do with Ktor. And agreed ^^

[Paulo Cereda]

This looks awesome, congrats! 🎉 Out of curiosity, any plans to extend the list of formats? Thanks, Alexander! gratitude thank you

(for context: we have a KMP font parsing library that fully parses OTF/TTF, but our WOFF/WOFF2 support is currently JVM-only due to zlib / brotli requirements. There's a project named Kompressor that relies on native libs to bring these two algorithms to KMP, but I was looking for a "lighter" solution)

[Stefan Oltmann]

I'm still waiting for the pure Kotlin brotli lib.
https://github.com/google/brotli/issues/1123

[Stefan Oltmann]

Indeed, the Kompressor lib does not target the web.
For https://stefan-oltmann.de/exif-viewer I'd need that.

[Alexander Hinze]

@Paulo Cereda Thank you 🙂 And yes! I am planning on adding support for both new compression algorithms, as well as more container formats. The next on my list are finishing up ZIP(64) support (excluding encryption extensions for now) and LZ4. But i thought about zstd and brotli too, among some others. I just can't give any ETA, i'll pick what people want most after i meet my own personal requirements.

[Alexander Hinze]

I don't know if i'd consider Kompress more lightweight than platform bindings.
My main goal is bug-for-bug compatibility and portability.
But if you want something that is pretty much always guaranteed to work, no matter the platform or Kotlin version/target, Kompress is a good choice i think if you are willing to invest a couple hundred kB more into binary size.

[Alexander Hinze]

Another thing i find nice about a pure Kotlin common approach, i can add support for new platforms super quickly 🙂

[Stefan Oltmann]

+1 for zstd. That's also quite interesting.

[Alexander Hinze]

@Stefan Oltmann noted! 😄

[Stefan Oltmann]

@Alexander Hinze Do you plan to support iOS as a target as well?

[Alexander Hinze]

@Stefan Oltmann It is already supported! ^^ Just no x64 apple target support because of 2.4+

[Alexander Hinze]

It should work on all Kotlin/Native targets, including Linux and Windows

[Stefan Oltmann]

Ah, okay. I didn't phase out x64 apple target just yet. You're quick with that. 😄

[Alexander Hinze]

Yes, because the x64 targets caused issues for me and our work team on 2.3.21 after they got deprecated :/

[Stefan Oltmann]

Ah, I see. So far it didn't give me trouble, so I did not change anything.

Until now, of course, because I like to use Kompress 😄

[Stefan Oltmann]

iosX64 is simulator on Intel Mac only. I don't think anyone works on such a device anymore.

[Alexander Hinze]

I can see if i can enable them later, but they caused some issues with commonization on our end for some reason.

[Stefan Oltmann]

Don't worry, I'll drop it. Nobody will miss that.

[Stefan Oltmann]

It's safer anyway, because to be honest: I didn't test that target in the last years. 🫣