https://kotlinlang.org logo
Join Slack
Powered by
Suppose for fun you redesigned how Kotlin librarie...
# library-development
j
Suppose for fun you redesigned how Kotlin libraries were distributed. What would you change?
🤔 1
Bazel has a feature called ijars, which is a .jar file that includes all the public API, and none of the implementation. It's a bit like a .h file in C, or the name.api file that’s managed by the Kotlin binary-compatibility-validator.
interesting 2
If an API-only artifact was a part of our library distribution, your build tool could compile against that alone
You can skip recompiles if the ijar is the only thing that changed. This is amazing and why Bazel does it
And for KMP, I could download only the ijar for all the platforms (say 
androidNativeX64
) that I never actually run locally
It's surprising that to compile OkHttp for 10 platforms I need to download 10 Okio jars. What if there was a single multiplatform artifact with code for all platforms? Like the macOS universal binary.
(Doing universal ijars is compelling. CI builds would run faster and cheaper!)
👏 1
I would love to see a certificate transparency-style public audit log of all published artifacts. If an attacker publishes a poisoned artifact to my artifact repo, the audit log makes it difficult to later hide their tracks by deleting the bad artifact
➕ 1
j
max 3 files: 1. public API 2. sources as I like to see sometimes how something is implemented 3. signing
j
What smart things do other ecosystems do?
@Javier and I think the binaries too
m
I want 
kotlinc
to be able to produce and consume some kind of multiplatform IR. No need for a static artifact for the public API, build tools should be able to transform the "full" artifact for compilation avoidance, like Gradle is doing. The reasonning is that in 90% of the cases, I will need the full artifact at the end to run my program so I'm ok to have only big one containing public API + implementation. This is more or less similar what Java bytecode is doing actually. Except we need this for KMP
👆 2
💯 5
👆🏾 1
➕ 1
c
+1 for having an API-only artifact, especially for plugins. Upgrading the version of a plugin when none of the configuration API has changed should not recompile all build scripts.
m
@CLOVIS no need to store that artifact in the repository. The full artifact contains everything already
c
Wouldn't it be faster to have it pre-generated?
m
You trade local CPU time versus Maven Central resources
👍 1
I think we should decentralize what we can
c
When I have some free time (ahah) I want to prototype feeding to the Kotlin compiler all libraries but without implementation (all methods replaced by a TODO()). I wonder how it affects the compile-time.
m
there is a case for api-only artifact though, if you're compiling a library then you don't need the full artifact at all, just the api-only one
c
To run the tests you need the full one
➕ 4
m
fair 👍
c
(my Linux CI runner can compile Apple targets but cannot run the tests, so having CI that only compiles is a thing that happens)
IMO the big question is how fast the compiler can 'extract' the public API from the full artifact. If it's anything but quasi-instant, I think it's worth having the API artifact.
m
There is also the case where the compile version is different from the runtime version. So you could save some resources there.
But it's hard to find the optimal point there. Is it worth using central immutable storage for this? It's all a tradeoff
If we're sticking with the current centralized model, I'd rather have central implement this as an optimization if they find the access pattern justify it (extract public API on the fly, a la Android splits) rather than making it a requirement and have everyone download twice as many files in (what I expect to be) most cases
Only Central knows the access patterns
v
> What smart things do other ecosystems do? Fantom has a very cool concept of pods. By having an artifactory as the first-class citizen, they have all-in-one identifier that can uniquely (globally!) identify a symbol by its name. So no classpath hell, very tooling-friendly (one can reasonably index packages that are not even in your dependencies, easy to auto-import and/or resolve, easy to have some global grep/find symbols etc.), versions baked into pods allow to do stricter (e.g. you can verify compatible version ranges for a "my-very-popular-but-sometimes-incompatible-logger-library") transitive dependencies management. I think the closest thing to this that is still alive is Go
🧠 1
👀 1
j
Maybe this is really stupid, but... It would be nice on my end if all common code shipped as IR and only the actuals shipped as separate platform code. I work on full frameworks across multiplatform and the compile times are getting absurd - it takes 45 minutes for CI to build these days, and the majority of that is the build/upload, not unit tests. It would probably slow down actual compilation on the target machine enough to be a problem.
Completely separately, it's always been my belief that a programming language should be tightly coupled with package management. If it is, the build system can be dramatically simplified in most cases, and if it isn't, it ends up tied to a package manager anyways. It's not like there's any significant use out there for Kotlin without Maven.
yes black 1
@Vsevolod Tolstopyatov [JB] Not sure how alive it is, but Unison (https://www.unison-lang.org/) has the coolest ideas on this IMO, and I think it's still alive, just... very very uncommon. It however, required dramatically different design to achieve that would never be applicable to Kotlin directly unless you guys were going to completely overhaul the language.
thank you color 1
c
Content-addressing makes very hard to implement 
api()
dependencies correctly and reliably, it's only really an option for 
implementation()
j
Very much so. In my personal design (I'm working on a language here and there) I did hash addressing on the API level only (function headers, interfaces, etc), but I'm also doing very very fine grained packages which really changes the dynamic too.
Unison's big deal was exact versioning and guarantees, which arguably aren't worth it precisely because of what you just said - the binary size explodes.
...and you have to constantly adapt between versions of datatypes. Eww.
c
I think for research languages / distributed computing it's great, because you're unlikely to have multiple libraries that depend on each other and expose each other's APIs. In a language like Kotlin with expansive frameworks like Ktor or Spring, that's not possible
j
Agreed, it would be hell for Kotlin - mostly just noting it since I think it's got interesting ideas to consider
👍 1
c
However, having an API dump and the ability to automatically detect whether an API is compatible with another would be very cool.
j
If you completely change the dynamic and do ultrafine packages, you get some other weird advantages too - the number of versions actually decreases because a large number of version bumps don't touch the majority of the library. But it's got other problems. Still thinking through that one in detail.
c
It's only safe if you know for sure that they will remain compatible together, so you have to very strict with your public API. Today, it's common that libraries make breaking changes in their inter-module API that's not meant to be usable outside
But yeah, KtMongo releases every module every time, but the 
:annotations
module has not changed a single time since it was created
j
Yep, you have to enforce how version numbers work if you do it. In our Kotlin libraries, we also always release our modules as a single versioned set. We have some libraries that depend on each other and it's always very irritating to push a major update through them all.
c
I think it's a tooling issue. I could somewhat easily publish just a single module, and handle internal version numbers, but that's a pain. Instead, maybe Gradle can detect (using the hashes) "I'm going to upload a version of this library that's exactly the same as a previous version, I'll just upload an alias instead"? That's kinda what Git does to avoid storing unchanged files in a commit.
@mbonnin I wonder if NMCP could do that by editing the .module to point to the JARs from the previous version. But then you would still need empty JARs because they're mandatory
j
Then you have a bazillion version numbers in the end projects though... I suppose though that's what version catalogs really are for, yeah?
c
No, projects would be released as one large set, exactly like today, but the tooling deduplicates the artifacts behind your back (so Gradle would know not to re-download the artifact, since it's an alias to a previous version)
j
Ahhh, that would be really nifty
c
That would be completely transparent for everyone, and decrease the storage size in the repositories
I wonder if it could be done with artifact transforms
I mean while we're there, we can go full content addressing of the .jar and .klib, and a "library version" is just the .module that contains just the hashes of the different artifacts
(that also solves the "klib is identical for iosx64 and macosx64" problem)
m
I wonder if NMCP could do that by editing the .module to point to the JARs from the previous version. But then you would still need empty JARs because they're mandatory
Probably could but I'm not sure the Kotlin world is ready for it 😄
c
and it allows distributing the final storage of the artifacts (companies could have a mirror of the actual artifacts, and the central repo is just the source of autority of which versions correspond to which hashes, that would make it wayy more lightweight)
(or even go full p2p and have dev machines distribute artifacts in the LAN)
add Tangled-style 'vouches' and you don't even need a central metadata repository
To make it immutable you would need the metadata repository to be append-only. Do you think I can still find investors by saying it's a blockchain in 2026?
m
Immutability is not that important if tooling implement metadata verification (which they should)
c
Immutability is very important, without it you don't know if the CI run is the same as your local run (welcome to the JS ecosystem)
m
This should be decentralized. Your build shouldn't trust the network for that but local verification data
c
One cheap to guarantee that is to force everyone to use hashes instead of dependency versions (or declare both), but I don't think the ecosystem is ready for that
m
Basically you burn verification data every time you update/add a dependency. This means you trust the publisher at that time. IF anything changes after that you get a build failure
If the artifact disappeared then you're out of luck 😄
But of course you have a mirror for your dependencies
c
Copy code
implementation("dev.opensavvy.ktmongo:bson-multiplatform:0.31.0:67ba53109667af9dbbf1a9dd84f1eb316c558a0a415afc89a09a923852cb8dc6d6ecc2e704abeb4abcb21d737f7c6919b9695cd00f8940489223ac0398360db1")
where the hash is for the .module (since it already contains the hashes of individual files)
m
That works
c
If the artifact disappeared then you're out of luck 😄
Right, but you can have a centralized MavenCentral-like backup repo. It only has to index the most popular libs, not everything
and any dev who ever pulled the lib can reupload it
m
the centralized repo should only have metadata + hashes / signatures imho
➕ 1
otherwise you're back to square one with having to spend lots of compute and bandwidth to ship binaries
c
So for
Copy code
implementation("dev.opensavvy.ktmongo:bson-multiplatform:0.31.0:67ba53109667af9dbbf1a9dd84f1eb316c558a0a415afc89a09a923852cb8dc6d6ecc2e704abeb4abcb21d737f7c6919b9695cd00f8940489223ac0398360db1")
then the resolution algo is: 1. 
GET <http://ktmongo.opensavvy.dev/.well-known/distributed-artifacts.json|ktmongo.opensavvy.dev/.well-known/distributed-artifacts.json>
→ get the URL of the primary artifact repository for this group (= client verification of authority via SSL). A library author can opt-in to use a shared repository like MavenCentral. 2. 
GET <http://ktmongo.opensavvy.dev/.well-known/distributed-artifacts.gpg|ktmongo.opensavvy.dev/.well-known/distributed-artifacts.gpg>
→ all artifacts must be signed by one of the keys in this file, otherwise they don't come from the library authors. 3. 
GET whatever-was-responded/dev.opensavvy.ktmongo/bson-multiplatform/0.31.0/artifacts.json
→ this file should match the hash & GPG signature, and contain the list of variants for that dependency, each with their own hash 4. 
GET whatever-was-responded/dev.opensavvy.ktmongo/bson-multiplatform/0.31.0/bson-multiplatform-0.31.0.jar
→ don't forget to check hashes & GPG signature The steps 3 and 4 can be performed to a proxying/caching mirror if you have one for your org.
➕ 1
Guarantees: • You get exactly the same artifacts that whoever added the dependency did, even for transitive dependencies • No need for a central authority, every client can verify the files themselves • Any HTTP proxying/caching proxy can be used as a mirror (I'm pretty sure that's a Caddy oneliner) • The version was actually uploaded by the library authors (or their key leaked, but that's already an existing risk with MavenCentral credentials) Impossible to implement: • Version ranges. But maybe that's a good thing actually.
I think that's solid? Can you break it?
Bonus point for "there's nothing specific to Kotlin in there", you could implement clients for Rust, JS, Python, Docker, etc and have a single repository per org for everything
r
Agree there's plenty of upsides to decentralizing a lot of that stuff, but one potential downside I see compared to what we have now with Maven Central is immutability. You can use a hash to confirm that the artifact you're getting from a random library is the correct one, but nothing stops the maintainer (or the maintainer's hosting provider) from taking that artifact offline and breaking your build. Would the idea be that everyone sets up their own mirrors/caches to avoid this?
m
nothing stops central from going offline as well - if you're worried about this you need to keep a copy of the artifact
c
Would the idea be that everyone sets up their own mirrors/caches to avoid this?
Maybe not everyone, but hopefully enough people that it's reliable. MavenCentral won't disappear, it can serve as a fallback. It would only need to index very popular libraries (and not necessarily everything), and it would get much less traffic, which would make it less expensive to maintain. Hopefully JetBrains would provide one as well.
What's crucial is that it should be very easy to setup a mirror. Currently it is not.
p
From my perspective having multiple per-target artifacts is not a problem per se, and it allows to download only what is needed (automatic deduplication would be good though). What I think would be very good though is requiring digital signatures much more prominently. This would make reliance on a single trusted source also less important). Maybe in line with what @CLOVIS proposed.
19 Views
Open in Slack
PreviousNext
Powered by