https://kotlinlang.org logo
Join Slack
Powered by
Hi everyone. Brian Fox from Sonatype / Maven Centr...
# library-development
b
Hi everyone. Brian Fox from Sonatype / Maven Central here. I’ve been in this slack for years it seems but was just reminded about it based on the publishing limits conversations. (https://central.sonatype.org/publish/maven-central-publishing-limits/) Lets consolidate the threads here.
➕ 3
kodee welcoming 3
👋🏾 2
kodee loving 1
👋 16
o
c
TL;DR on why Kotlin Multiplatform is specifically affected. Whereas a typical Java library releases one JAR per module, Kotlin Multiplatform libraries generate one JAR (or one KLIB) for each platform they support, plus some metadata. Since there are ~15 popular platforms, the limits are essentially ×10 smaller for us than they are for the typical Java library. I think that's the main reason why so many more of us are above the threshold, compared to the average Java maintainer.
✅ 2
👍 5
b
Ok. So this takes my thought process in 2 parallel directions:
m
Note that JetBrains has/had plans to reduce the number of files with uklibs (here) but this effort seems mostly halted right now
➕ 3
s
KMP is definitely worse off than most, but even plain Kotlin/JVM with platform-native helpers has 5x the usage (win x64, win arm64, mac arm64, linux x64, linux arm64). I did one small release of my small library with a bugfix and I am already ~50% there on the files count (Kotlin/JVM + native helpers + a couple optional artifactIds). Yes, it's OSS so it will be excluded, but even then these limits are quite unrealistic...
☝️ 1
a
It'd help if the required files (checksums, signature files, sources.jar, javadoc.jar) didn't count towards the limit. A 
javadoc.jar
is required - but it doesn't make sense for Kotlin, and those files are usually empty.
➖ 1
b
Ok. So this takes my thought process in 2 parallel directions: 1. does it need to be so large, what can be done to make this make more sense for the ecosystem. I’ve had similar conversations with the React Native teams where they were doing things in ways that were “unexpected” 2. Separately, and knowing that will take time, how can we adjust limits so that this more or less just works and every KMP doesn’t have to request an exemption. I don’t want to process thousands of tickets any more than you all want to do 1.
➕ 5
c
(additionally, there is a Gradle bug which causes hash files to be generated twice for no reason, which nearly ×2 the total number of files 😕)
☝️ 2
👍 2
😐 2
b
Don’t get me started with Gradle. That tooling creates tons of wasted consumption and I’ve told them about it for years.
➕ 3
❓ 5
TBH, I never really liked the file count metric since I first described it to the team.
💯 2
👍 1
c
cc @Joffrey (head of the ex-Amper project / Kotlin Toolchain)
thank you color 1
b
Its useful to have some upper bounds though because some projects to crazy stuff and we want to have a reasonable inspection point to say “is this right, is this the best way to do this?”
m
Removing the file metric/increasing it (a lot) would surely help
r
For path 2, it should be straightforward to detect that a project is KMP, if you're open to auto-detecting that and setting different limits
➕ 5
b
That’s one thing I wanted to ask about yes
That might be the easiest way to address it. Figure out what “normal” looks like for KMP and just automatically use that
if I were querying my system for all KMP projects, what should i look for?
c
To detect KMP: Every KMP project (at least currently) has a 
.module
file which contains
Copy code
"org.jetbrains.kotlin.platform.type": "native"
where the right side of the string is NOT 
"jvm"
For example https://repo1.maven.org/maven2/dev/opensavvy/spine/spine-server-linuxx64/0.2.0/spine-server-linuxx64-0.2.0.module
➕ 2
k
image_Original.jpg
This is my example - one real artifact - the jar file. And 49 things that get created from the process.
b
Yeah that’s lame ^^
They should fix that.
k
Now multiply this by about 10 modules in my library, and just one release is over the quota.
💯 1
b
Let’s be honest, a whole bunch of this process is going to require tools to do things that were unimportant to them, and we’re going to need user peer pressure to make them do it.
➕ 9
k
I do two releases a year. But with these limits I can't even do one full release.
b
It’s all sprawl and waste. It’s all wasting the valuable open source dollars that we get across the board.
➕ 3
👀 1
h
Almost all KMP libraries use Gradle as build tool, uploading the module metafile.
➕ 2
☝️ 1
b
@brian.demers fix it.
ah. wrong Brian, sorry
k
Not sure how you see this playing out. That developers who can't publish on Maven Central anymore form some sort of a collective pressure on... who? Gradle? Jetbrains? Somebody else?
b
If there is something a particular tooling is doing that causes people to be more likely to hit particular limits, yes, those users should push back on the tool. I have the exact same conversation with companies that get rate limited. When they come back and tell me that it’s impossible for them to make their Gradle builds go through their caching proxy, I point them at the ticket and tell them to go tell Gradle it’s important for you to get fixed. I tell Gradle their users are overrepresented in the people who are getting rate limited, and there’s not much more I can do about it.
But that’s more of an aside here. We’ll deal with that separately.
Part of this process of trying to rationalize all the sprawl across all these different dimensions is simultaneously recognizing that we have to start with limits that recognize where the ecosystem is. Because my goal is not to blow up everything, but also not to ignore all of the not ideal stuff that I discover in the process and find a way to get that changed for the long term.
Back to how to identify KMP builds, is there a file name or folder pattern I can look for that doesn’t require me to also read inside the file looking for !jdk ?
k
I feel that the core of the issue here for "true" open source developers and maintainers is that the limits on file counts and release counts is way too low on monthly basis. I have three related projects that have coordinated releases, twice a year. With these limits I can't even do one release. Just one. And if there's a bad bug that requires a quick 
.0.1
followup - well, that's too bad.
👍 1
b
I hear the frustration with suboptimal publishing tools, but consider that the pain you're going to impose this way won't move them meaningfully, and will damage the Java OSS ecosystem if not finessed "just so". Limits are very obviously tricky to get right, and I don't think we're there yet. It's not even clear to me, having read blog posts and announcements and such, what your real drivers actually are. Is it egress costs? Storage costs? What's the deal? There are many dimensions, but some must matter more.
☝️ 5
☝🏾 2
a
They should fix that.
@Brian Fox Do you mean Gradle should fix the number of files published? Aren't most of them are there because of Sonatype requirements, and it's not because of Gradle?
💯 5
c
The 
.module
trick is the only reliable way, because that's what the tooling itself uses. If you want a quick-and-dirty way, KMP leads to having multiple modules that have nearly the same name, but with a platform prefix.
Copy code
foo/pom.xml
foo-jvm/pom.xml
foo-js/pom.xml
foo-linuxx64/pom.xml
etc
m
leads to having multiple modules that have nearly the same name, but with a platform prefix.
I'm a bit wary about this because some folks may release artifacts separately (from different CI, etc...), I like the 
.module
trick better
b
Let me address that @Kirill Grouchnikov It’s a 3 month average so that’s unlikely to happen anyway unless you did it 3 times in a row. Second, the most likely scenario is that the release is simply slowed down and held for a time, not fully blocked. We are still implementing the hard limits implementation but we recognize that simply closing the doors isn’t always an option.
➕ 2
t
that screenshot was: 1. a jar 2. sources jar (sonatype requirement) 3. javadoc jar (sonatype requirement) 4. pom metadata 5. gradle module metadata 6. signature files for the above what does "fix it" mean in this context?
👍 1
➕ 3
c
Aren't most of them are there because of Sonatype requirements, and it's not because of Gradle?
Not really. All the .asc.shaXX are useless. Of the 4 shas generated, only 2 are mandatory. The only file that's structural that Maven doesn't have is the .module
m
aren't the mandatory checksums ancient?
☝️ 1
☝🏾 1
b
some of them yes
updating the validations to make sense for eg Kotlin is on the roadmap.
m
so we should get rid of the safer newer ones because of bloat?
t
you require the oldest/worst security checksums and don't support newer/better checksums
💯 2
☝️ 3
☝🏾 1
b
i actually want to open those validation rules so the community can help us
c
>> leads to having multiple modules that have nearly the same name, but with a platform prefix. > I'm a bit wary about this because some folks may release artifacts separately (from different CI, etc...), I like the 
.module
trick better Right, it's not a good long-term solution. But if @Brian Fox wants to do a quick regex filter right now to get an idea of how many modules that is, it should be good enough?
👍 1
r
I have such files in my projects but I'm not sure where they came from: https://repo1.maven.org/maven2/dev/kilua/kilua/0.0.35/kilua-0.0.35-kotlin-tooling-metadata.json
b
yeah, i was looking for a quick way to do some data analysis, not the final inspection implementation
a
Hey there, Anton Makeev here, KMP project lead at JetBrains. We are looking at the options right now how to avoid that in future - that’s a good point about publishing multiple artifacts. Let’s see how what options are available short-term. @Brian Fox We have info on how to distinguish KMP libs ( @Azat Abdullin could you help with the KMP identity question?). Would you be able to check how many are affected by the limits?
👍 1
m
TBH, we could probably remove the sources + javadocs requirements too since some projects send empty jars there
😑 1
agree 1
🚫 2
k
Paging @Gabriel Ittner as the main developer of https://github.com/vanniktech/gradle-maven-publish-plugin
b
^^ yeah, a bunch of commercial orgs using me as a free cdn.
they put empty jars
i will need to clamp down on that
blob thinking upside down 1
m
I like that it's a nudge to publish sources but it's no real enforcement
c
(while we're having eyes on this: if you are a library developer and want to have your Kotlin documentation in the generated javadoc.jar but you don't know how, DM me and I'll show you, I know it's not obvious) I know multiple tutorials use the empty Javadoc JAR just because they're too lazy to set it up correctly
n
At klibs.io we use 
*-kotlin-tooling-metadata.json
filter to search for modern KMP libraries.
b
If you had to rank the cost drivers, what matters most? File count? Bytes inbound? Metadata management? Egress costs? Storage?
b
It’s all of it. The surface expands quadratically when you look at this
a
We can detect the platform-specific KMP publications by filtering them by 
packaging: klib
. And then we can parse 
component
section of the Gradle 
.module
files to detect the main publication that they are linked to
*-kotlin-tooling-metadata.json
also works, but not all publications have it
b
does having extra hash files that no one downloads matter? Not really. Does having 10 billion wasted buckets and things to index and data analyze? yes.
It’s like Carbon and Carbon tax. Does a campfire matter? No. Does burning down the rainforest? Yes.
k
Who is requiring nine signature / checksum files for each artifact?
b
No one cares about the waste created until you start creating limits
➕ 1
k
[Certainly not I as the original developer]
m
@Azat Abdullin is 
*-kotlin-tooling-metadata.json
present when using "recent" versions of KGP? If yes, I think it's a reasonable ask for people to update
b
The extra files is really a red herring for the publishing limits though. I tried to explain it in the faq: Why does file count include signatures, checksums, source jars, javadocs, and related files? Because the thresholds are derived from actual Maven Central publishing activity, the inclusion of these files should not affect any project disproportionately. Every publisher produces signatures, checksums, POMs, source jars, and javadocs — these files are part of what everyone publishes by definition, so they are already reflected in the usage data the thresholds are based on. The ratio of supporting files to primary artifacts is roughly consistent across the ecosystem, meaning the resulting limits are proportional regardless of how you count. Counting only primary artifacts would also introduce ambiguity that is difficult to resolve consistently. It can be very hard to distinguish a main artifact from an attached artifact, and Maven Central supports many different packaging systems with different conventions. Files are straightforward to count, and because the limits are derived from real usage patterns rather than an arbitrary multiplier, the metric works fairly even if the raw number looks large.
➕ 1
k
Sources / javadoc requirement is also not from me as a developer, and it ends up 3x'ing the cumulative size of each release.
💯 1
b
yes but it should come out in the wash ^^
b
So far I'm mostly hearing you talk about file counts and signature files - let's say we fix that, and only require sha256 or whatever. Would that solve problems? "Easy to count" implies that it is difficult to quantify the real problems.
k
It's absolutely not a red herring. The real thing is one file. Now there are 50x files with all of these extra, and the total binary size is 3x.
All counting towards the quota
b
eg we didn’t pick the number based on what we think people should do, we did it based on analyzing the norm.
yes, well this is why we are trying to understand why Kotlin is different 😉
k
My numbers are not for Kotlin or KMP. It's a pure Java library. One jar, 49 extra files around it that I honestly don't care about.
b
I think we’re losing sight of the fact that my intent for the limits was that this number of certain different dimensions should trigger an inspection. It was never intended to be a limit for typical open source projects.
👀 1
👍 1
b
I think any kind of multi-module JVM project is going to run in to these limits fwiw. My most-used OSS library is now Kotlin, but even when it was Java and maybe five Maven modules, it would have exceeded these limits.
b
Except they weren’t. I was talking to some prominent Maven developers yesterday, and they showed me their dashboard, and they were all well below the limits.
c
It was never intended to be a limit for typical open source projects.
I think I'm not the only one who didn't get that part of the announcement.
➕ 4
m
Has been said before but Kotlin has no package visibility so multiple modules are encouraged.
➕ 2
Bundle that with multiple platforms
b
@CLOVIS yes. it wasn’t clear enough. I rewrote the docs a few times, and we even just changed the working on the usage center itself to set a better context
➕ 1
t
No one cares about the waste created until you start creating limits
comments like this make me wonder if you realize that the burden for this is actually falling on open source developers. That is, people putting in countless hours for free providing services to other devs. I get you're also providing a free service, but pushing the cost to us in the idea (??) that we'll somehow pressure the toolmakers to change their ways is really backwards
👍 1
b
My fault. I reviewed it, but I was focused on other things that needed to be fixed. It could have been so much worse.
m
Apollo Kotlin has 15 modules. Every Most modules have 14 targets.
Times 50 => ~10k files
👍 1
b
@tony I’ve spent almost every hour of my free time, weeks, weekends, and nights for the last six months dealing with this in different contexts. I think I understand it.
m
It's not huge in bytes. But huge in files
k
Copy code
I was talking to some prominent Maven developers yesterday, and they showed me their dashboard, and they were all well below the limits.
Can you link to one of these to see how many files they have for one of their recent releases on Maven Central?
b
@tony But also, this isn’t meant to create a burden for you. That’s why I’m here, trying to figure it out.
a
To specifically answer why Kotlin Multiplatform is different than a Java (or pure Kotlin JVM project), each KMP target has a separate published module. So I might have one Gradle subproject. If it was Java, then that's one 
.jar
(+ other required files). But if it's a KMP target with 10 Kotlin targets, then it's ten times the number of files - no matter how big the codebase. Why does KMP publish separate modules per target? Long story...
🙏 2
a
@mbonnin I will check with the build tools team about the tooling metadata. @Nikita Vlaev Maybe you can help?
🙏 1
👌 1
b
image.png
This is a very active maven project creating lots of tooling around Maven.
I published the updated numbers on the docs last night, i feel like people still think I’m making this all up…
image.png
m
Are we still discussing why KMP projects are larger in number of files than JVM only? Because that answer has been given here and here
➕ 1
c
I don't think your numbers are false, but mine are very different and I don't think I'm doing anything weird
➕ 2
b
@mbonnin no i get it now. I’m trying to figure out what to do about it. What I think is that even if we set the numbers at the top 1% it still would trigger on KMP. Which is a moment to ponder but we’ll deal with it
👍 1
m
Make an exception for KMP using the detection tricks from @Azat Abdullin and discuss with Gradle and JetBrains about how to improve this in parallel?
👆 1
c
I think if you set to the 1%, it will still trigger for some KMP libraries, but so many less that it becomes manually reviewable.
m
Oh, and remove the requirements for checksums
My hunch is Gradle fixes are probably low hanging fruit. I can probably filter in nmcp as well
➕ 2
The uklib a.k.a multiple targets discussion on the other hand is ... much longer
k
Thanks for the link. Looking at https://repo1.maven.org/maven2/eu/maveniverse/maven/plugins/toolbox/0.15.14/ they publish 5 checksum files per "real" file, compared to 9 from my libraries.
And no 
.module
😁 1
So if this is one of your examples, it's already running afoul of the release limit
b
No because it’s a 90 day average
k
And if that's their pace of development, they won't be able to do that
b
The way I approached this for consumption limiting was clearly better by starting super high and slowly bringing it down. But we don’t have to publish those numbers so it’s a little easier to manage.
➕ 1
k
Not sure what Takari is but https://repo1.maven.org/maven2/io/takari/takari/73/ doesn't look like a real release
b
We didn’t want to make publishing feel like a moving target. And yet, here we are.
That looks like their parent pom.
b
The difficulty I'm having is that, after all this discussion, it's still not clear what is going to meaningfully move the needle for you. We've talked a lot about file counts, and about how those aren't really what matter but it's easy to measure. Your business is your business, of course, but it's hard for us to be helpful if we don't understand the problem explicitly. As it stands, this feels like you ought to sit down with someone from JetBrains and someone from Gradle.
➕ 7
r
You have all the data. You know which projects exceed the limits. Each project has a pom file with a "license" field. Perhaps it would be easier to automatically recognize open source libraries that meet your requirements? As long as we can determine what those requirements are?
b
Sadly no.
Open washing is a thing.
k
I would also suggest, if I may, to look at more prominent Java projects that are a bit more used in the ecosystem. For example, https://repo1.maven.org/maven2/com/squareup/okhttp3/ and its latest 5.4.0 release - how many files, the binary size, etc.
t
@Azat Abdullin is 
*-kotlin-tooling-metadata.json
present when using "recent" versions of KGP? If yes, I think it's a reasonable ask for people to update
Yes, this file is always published with recent Kotlin Gradle Plugin versions for KMP. We even discussed idea of publishing it for Kotlin/JVM projects, but for now put it on pause.
❤️ 1
thank you color 1
r
I can't believe you are ready to read all these (thousands?) emails with manual descriptions, manually browse GitHub repos and webpages to decide if the someone is "good OS" or "bad OS" developer 🙂
Unless the number of affected projects is not so big ...
b
That was the intent.
(not so big)
Also, many can be pre-emptively waived. This was planned before any actual limits take place. But it was a mistake not to do it before soft launch clearly.
Also, we knew we might have to slip the date to implement the limits as we learned, like is happening here. But we learned last year in the old OSSRH migration, people ignore you until a date is imminent and we needed to get the ball rolling. We accidentally created too much rolling.
Thankfully I held the email announcements to all publishers or it would have been 1000x worse. This gives me time to fix the docs, fix the bugs, adjust the numbers etc.
m
@Brian Fox very pragmatically, can checksum requirements be removed altogether ? That would give a escape hatch to dramatically reduce the number of files with little to no impact I believe.
r
Too much red color in the usage center 😉
😂 1
b
I updated it to be less red.
😄 1
@mbonnin maybe. We could stop counting those all together, but then the number would come down proportionately. I still think it’s reasonable to have an upper limit inspection point to catch broken things.
Well, I have what I need to do the data research on how to solve the KMP problem.
r
If it's not clear from the message above, filtering on the presence of a 
*-kotlin-tooling-metadata.json
file should be a reliable way to detect active KMP projects. Does that give you enough information to look into whether you can set alternate limits?
b
Are there other questions I can answer? Like have I completely lost my mind? I don’t think so, but can see why if folks haven’t followed along on the open letters and sustainbility push, they was a surprise.
b
To reiterate Brian, I really think you should set up a meeting with JetBrains and Gradle to actually Solve The KMP Problem. Dialing in limits seems lower-leverage.
b
Agree. They have already reached out.
💯 4
👍 1
👍🏾 1
Parallel paths. Goal 1, make this make sense now. Goal 2, reduce waste
m
This is a surprise because most people here aren't huge users but regular OSS maintainers, not because people didn't read the posts
I did appreciate the blog posts but didn't expect to be that impacted
Also appreciate the awesomeness of having Central in the ecosystem
a
@Brian Fox Please also look at removing the javadoc.jar requirement. Usually it doesn't make sense for Kotlin (the source.jar has the same info) but some libraries do use it to share the Dokka documentation. Even if a javadoc.jar is useful, then it's not required for every KMP variant. If a KMP subproject has 10 KMP targets - That's 10 javadoc.jars, 10 .asc files, and 40 checksum files. But, if using Dokka, then just 1 javadoc.jar would contain all information for all targets.
☝️ 1
➕ 5
☝🏾 1
☝🏻 1
b
Yes I know. I’ve wanted to fix this for a long time. Making you create an empty jar is stupid.
thank you color 5
@mbonnin “I did appreciate the blog posts but didn’t expect to be that impacted”. -> I wasn’t expecting this level of impact either 😉
j
Hi Brian! I'm not sure I understand how you couldn't know about the impact. I thought you had looked at real life data and decided on the cutoff point based on the data. So you must have known exactly how many projects would be impacted by the quotas, or did I miss something?
j
requiring md5 checksum is also useless, gradle will only read sha1, and maven checks sha1 first and falls back to md5, so requiring both is pointless
b
Fair question. I didn’t realize that basically every KMP project would be affected from go. Also as I mentioned elsewhere, I expectd to be able to pre-process a bunch of the limits and we just didn’t get that done before this rollout. I knew it was required before actual limits.
j
Yeah the problem with setting the quota based on some average is that you don't see groups that have very different averages (like the KMP group). Maybe a progressive roll out would have helped here. Applying the quota only on some select days for a transition period.
b
Not an excuse, but some context, I have been deeply involved in a lot of the industry prep for Mythos over the last month, some things that aren’t fully announced yet, and that diverted my attention from really deeply inspecting everything. I reviewed all the stuff, fixed a bunch of things, but really expected more time to get everything in place, expecting that most didn’t matter until limits actually took effect.
kodee loving 1
We’ve also been really successful in rolling out rate limits without blow back and I probably got lulled by that too.
(consumption limits)
j
Thanks for owning the "mistake", Brian, I appreciate it. Now, I have to say I've always been amazed by how Maven Central could be something so big and free for so many decades. Thank you for this! I do understand that it is not sustainable to have unlimited publishing for all these people, and that you have to put some sort of safeguard in place, so I don't blame you for this
➕ 10
b
Thank you.
j
I do have to say I'm a bit disappointed with how the transitions play out, though. First, the consumption limit, which heavily disrupted our company's CI, and broke all our builds just days before KotlinConf (where things get really intense). Now, this publication quota :/ but at least you guys have been quite helpful in a reactive manner, so there is this 😊
b
And, there’s the blow back 😉
😄 1
But yes, I was basically monitoring and responding to all that stuff in realtime. Like until 1am most days and weekend.
j
Sad to hear you had to do that! (But thanks for doing it) I just hope next time will be more incremental 🙏🏼
b
Well the rate limits where. Sadly most people ignore the limits (which increase block length slowly) until they are 10+ hours long. So for them it feels sudden. For us, it’s like, we’ve been blinking your lights for 2 weeks now…
Every time. Without question.
j
I have many personal OSS libraries that I don't release every week (maybe not even every month), so I have no opportunity to see such thing coming if it's only rolling over 2 weeks
b
sorry, i meant the consumption limits
j
Ah my bad, misunderstood 😅
In any case, thanks for trying to find solutions for KMP publications 🙏🏼
b
It was true in jetbrains also. I count 12 actions with increasing length of time from 1-2-4-8-14 and more hours before we heard from anyone. And remember, these are usually just ips, we don’t knwo who they are until they reach out and give us an egress ip to investigate
But when people reach out, explaining that is usually not part of the narrative because by then it’s water under the bridge and it happens to everybody. However, it’s the unfortunate reality that there is a long. of time where the limits are being triggered and nobody notices. By the time they notice it, it’s a panic situation, which is why I end up having to monitor this in real time and try to respond as quickly as I can because my goal is not to end people’s worlds. My goal is simply to get the behavior under control.
➕ 1
Also, i’m sure a bunch of folks here have submitted exception requests. I’ve processed some of them, but until we get the more systemic things fixed for KMP, I’m going to focus there.
j
i made a PR to the popular gradle plugin for publishing to MC that reduces the number of files published by about 60%. Dropping the requirement for MD5 would push it down even more
❤️ 1
🚀 5
🙌 4
b
Ok, I have some preliminary numbers. We used a few of the heuristics above to find about 3700 KMP namespaces. Analysing that cohort the same way as we did globally produces the following:
Copy code
Percentile    Release Count Average    File Count Average    Total Release Size Average (MB)
90th          9                        7,153                 144
thank you color 4
👍 1
How does this feel based on what folks are seeing. Keep in mind, we would expect some to be above this of course
For comparison, that means KMP releases are in the top 2.5% on file count, 7% on size and release count
thank you color 1
k
Another aspect of this heuristic would be that, relative to other parts of the Java ecosystem, these are freshly new and actively developed. Cutting them off from the free tier is going to get a lot of pushback.
m
how much overlap is between these metrics? e.g. what's the percentile that violates any of them?
b
The global stats are only things published in the last 90days anyway @Kirill Grouchnikov
so not stale
j
Shoot, I'll need to cut a release of my OSS stuff soon then to be part of the pool :)
z
the limits still do sort of punish more active developers/projects no? Which feels like it would have an unintended negative effect
those new numbers are still incompatible with the projects I'm involved in, at a glance. • slack is two releases of github.com/slackhq/circuit • zacsweers is 7 releases of just 3-4 projects • Uber (I believe) is just two releases of github.com/uber/nullaway ◦ This is the closest to doable but this is also a single JVM artifact, no KMP
a
@Brian Fox thank you for the estimation! that confirms our intuition and shows even higher average/number of affected libraries than we expected. Given so broad impact, we are right now looking for the long-term solutions for the community: • using file compression for the klib binaries. we have early prototypes already that depending on the compression ~3x or even 30x improvement. this needs further experiments and balancing size/compute. • packing multiple klibs, sources and javadocs together to minimize the amount of files. Have some early designs
b
Hi @Anton Makeev / @Zac Sweers just a follow up. I’m looking at the data and available options on our side. Today is a holiday for our US team but next week I’ve already put this on the list for us to brainstorm what to do. We will figure out how to make it work. I don’t want to have to manually approve a bunch of stuff for nothing. I’m just not sure yet the right mechanism. It could be we find these and just insert into the system higher limits (we can already do this per namespace or per org) or we build it into the system to make it dynamic by recognizing different projects types. Bottom line here, if you are an open source project and not part of a commercial GTM engine, there’s nothing to worry about. I’m here because I want to make this work as smoothly as possible. But putting in some limits is a good thing. It’s already driving some of the rationalization we need, like finding and removing the wasteful hashes, thinking more carefully about size etc. We’ve also had a handful of really large clearly commercial publishers reach out and engage us directly.
thank you color 3
💜 13
b
No intent to spam here and I fully understand where this is coming from. Still, some tangible context based on a concrete release to put things into perspective. • jvm-only publications for one of our libraries is 15 MB (including signatures, docs, sources) • All other KMP targets (sans JVM) for the very same library is 435MB (including signatures, docs, sources) Since we try to support all KMP targets with everything we publish, this is a 30 times blowup. I've combed through the release artefacts and it's really interesting: • A LOT of that overhead comes from the 
javadoc
jars, which contain 99% the same for all targets. • 
iosSimulatorArm64
is functionally 100% the same as 
iosarm64
evertything that is not hashes and klib are, 100% identical and there is nothing to be done about it the way kmp publishing works. The only thing we're guilty of that is not default is publishing version catalogs and SBOMs alongside of our artefacts, but those were included in the metrics from before (and don't do anything meaningful for the per-target numbers). @Anton Makeev obviously, compression would help A LOT, but of course only if it takes all raw data across all targets into account so it compresses before creating per-target jars for everything. LZMA-compressing the full 450mb as they were got it down to 168MB. I believe this could be at least halved, when done properly and systematically. Again, I get that the current situation is not sustainable for maven central because it produces massive load, but this extreme blowup might push some devs to intentionally not publish for some less popular KMP targets, or skimp on API docs, which would be a very undesirable outcome.
👍 1
b
Wow that is eye opening @Bernd Prünster. Can you point me at the central url for this so I can visualize myself the comparison between the jvm only and the kmp?
A 30x explosion in size is a lot of waste. Multiply that across the ecosystem and that’s burn down the rainforrest kind of waste.
➕ 2
b
yup, not sustainable. Compression alone would cut that down to a quarter, probably. smart deduplication would then probalby again shave off yet another quarter of that
but that is an utterly unqualified gut feeling. I did take two information theory lectures, but only to realise I'm not smart enough to carry on
b
what is the coordinate in central?
b
There you go: https://mvnrepository.com/artifact/at.asitplus.signum. Compare one of the 
indispensable
or the 
indisepnsable
artefacts for jvm, vs all other targets. I'd be curious if you find out whether • we actually do something funky during publishing • I was sloppy when digging through the local publications • the 30x explosion is actually real
b
mvnrepository.com is a 3rd party that just sells ads fwiw. But I can find it from here 😉
b
but i only found concrete artefacts and not a group overview an central.sonatype.com
b
That’s true
Looking at: https://repo1.maven.org/maven2/at/asitplus/signum/indispensable/3.24.0/ what I see are redundant hashes of the .asc that could be nuked, but pretty normal otherwise
s
Honestly I know mvnrepository is a 3p but it's pretty great, despite the ads. Y'all could copy a few of their features back (like in this case). Although they index other useful repos too, at least for some niches, which has value in and on itself 😁 but fair to say, the newer Sonatype website is a lot better than the older one was and since it launched I actually use it more than I used to
b
That’s progress at least 😉
💯 1
s
</ot>
b
I see now. @Bernd Prünster This is kind of shocking in the repetition here. For something like Jdk that is supposed to be run anywhere, why do we even need to ship the binaries in like 15 different forms?
m
@Brian Fox KMP is not only running on the JDK
☝️ 4
It's running on native and a bunch of other platforms
s
There are use cases where you need native helpers too, even if you're not KMP
👍 1
b
Ok well still. It seems like the maven classifiers could have been used for dealing with the multiple packaging, but I suppose the pom/dependencies might be different. Still this really needs a much closer look in design for repository efficiency.
b
The prime issue is docs replication, imho. Indispensable contains virtually no target-specific Foo and the docs are still 5mb per target
m
Removing the javadocs requirement would indeed be quite useful. @Adam S also mentioned it there
v
Hey folks, from the Kotlin team side, we'll be using https://youtrack.jetbrains.com/issue/KT-87203/ to keep you posted about the state, results of the investigation and changes on our side.
🙏 3
🙏🏻 1
kodee loving 2
⭐ 1
a
@Bernd Prünster I'm curious - what is the use-case for uploading the generated Dokka HTML in javadoc.jar? Do you know if anyone actually ever downloads the javadoc.jar to view the Dokka HTML page? My firm opinion: Dokka HTML should not be uploaded to Maven repos. The sources.jar effectively contain the same information. And the HTML docs are not that useful: Dokka HTML doesn't work properly without a server, so even if someone did download it they'd need to run a local server to host it (which is easy, but at that point just extract the sources.jar and look at them in an IDE?) Instead, I think it's best to host the docs online, and just put a single 
readme.txt
file in the javadoc.jar that says "This JAR is intentionally empty. For documentation, see $linkToDocs." And I hope in the future Maven Central relaxes the Javadoc JAR requirement.
m
@Adam S one use case is using https://javadoc.io/ to save the lib maintainers of hosting the docs and have a central docs website
Another use case is to use Maven Central as source of truth for kdoc. I use this to regenerate multi-version kdoc without having to maintain a separate state.
The former isn't too useful I'd say but the later is.
Of course no need to upload kdoc for every module/target. Just the aggregate kdoc is enough
b
@Adam S in our case, we can almost certainly get rid of it, as we host the HTML docs elsewhere. For us the only real thought was: Maven central requires this. Let's be good citizens and attach the actual docs to every artefact as per the requirements. The only important user-facing feature we require is that API docs remain available, but this should be covered with the sources jar. In general, though, this may vary for others as @mbonnin mentioned and uploading a dummy javadoc jar just to pacify the maven central checks is a red flag that should lead to relaxing this requirement, rather than working around it.
👍 3
m
Agreed. And to be 100% clear, I completely support removing the javadoc requirements. Despite uploading some of them myself, I don't think this should be a hard requirement.
j
i would be hesitant to remove javadoc publishing unless some changes were made in how gradle sync works. if a published library has no gradle module metadata, intellij (or maybe just studio? there's customizations that dont match between them) will ask gradle to resolve javadoc and sources jars. If those dont exist a 404 is usually more expensive because it would be requested every time (unless caching is applied). also if you are pulling through an artifactory mirror that is a virtual repo of N upstreams, the request for javadoc.jar will search each upstream and the overall request is quite slow. IMO this is a bad decision inside the IDE but worth calling out
t
IDEA does not use javadocs - all documentation/quick references are pulled from sources
☝️ 1
👍 1
m
very pragmatically, can checksum requirements be removed altogether ? That would give a escape hatch to dramatically reduce the number of files with little to no impact I believe.
I'm taking that back in light of this comment. There is an impact to removing the checksums for 3rd party proxies who want to validate the contents of a file without downloading it all. Still, publishing 4 checksums feels overkill (especially when files are a metric). Ideally, I'd love that computation to be made by Central, i.e. compute the checksum based on the jar that is uploaded. That would allow to be more flexible in the future in deciding what is a good checksum to send without having to update all publishers. If that's not possible, changing the requirement so that a single 
.sha512
file passes would be nice.
b
Good feedback.
🙏 1
s
One more bit to the discussion about checksums is that when I used to run package search at JetBrains, one thing we noticed is that a lot of checksums are missing because historically they weren't required or they don't match the binaries when you download them, especially for older libraries. So, having central compute them would be absolutely great.
💯 2
j
Yes, we also have special handling in the Kotlin Toolchain's dependency resolution because some checksums are just incorrect
b
Maybe HTTP's Content-Digest header could be used instead - although I imagine this would break every tool that expects these files to be there.
interesting 2
👍 2
j
This sounds exactly like what we should use, indeed. Well making checksums optional would also break tools that expect the checksums to be there. I still think it would be worth making the ecosystem more modern in this respect.
👍 1
Now, when thinking about consuming tools, we have to keep in mind that Maven Central is not the only Maven repository out there. That said, maybe Central is the only one enforcing md5 and sha1 checksums, so maybe all the tools have to handle the case of missing checksums already. In any case, I think new modern features like this could be offered as an option on Maven Central. Depending on the tooling landscape, keeping compatibility might just amount to publish some checksums, but not necessarily md5/sha1.
➕ 1
p
@Vsevolod Tolstopyatov [JB] @Brian Fox While perhaps a bit late I'd also want to put in my 2 cents. I maintain the xmlutil library that works with kotlinx.serialization to support serialization to XML. It supports (or attempts to support) all platforms supported by kotlinx.serialization. There are also multiple sub-modules (some more important than others). After an initial release that blew through the file limits (10000+) the next release was better (I have my own upload, so I filtered the creation of the uploaded zip to not include those hashes of the signature). I fully understand and support the desire to limit/reduce resource usage, especially when those things are just wasteful clutter that use up resources for no benefit to anyone. Collaboration and coordination (with some pressure) seems to be the way to go. Anyway I do have a wishlist of things that would make it much easier to do: • Documented best practice (including a lists of files that can be safely removed - e.g. maybe hashes for tooling files) • A single source file per module (or even distribution) • For KMP javadoc seems not relevant (I'm not sure it even works in dokka), but in any case a single javadoc/dokka per distribution (the way I generate it for github.io) is best, per-logical module would work, per target makes no sense. • Good defaults for automatic publishing of multiplatform artefacts (although my library predates a lot of things so there may be cruft in the build system)
j
To add to the discussion, and I just discovered about the limits: With Maryk, I have 102 MB with 14,310 files within a single release. It is fully open source project. And the size is that big because of multiple modules with 23 variants (targets from linux, macos, wasm, js, android and other targets) So these limits make it difficult to work with. central.sonatype.com/artifact/io.maryk/maryk-core I hope a solution can be found.
a
I will have troubles with Kore too because when I'm working on Kore I'm releasing like 10 releases per week because I create 1 small release per Minecraft snapshot (when they are in the development cycle of any versions they release 1 to 2 snapshots per week, then multiple pre-releases and multiple release candidates with often just 1 or 2 fixed bugs) which means I'm releasing a version of Kore with no direct code changes but changes in the generated files (parsing Minecraft Metadata and generating enums and consts) In the last weeks I've even released up to 14 releases in the same week, with around 1200 files per release and 8mb The new limits are far too low for my pace of working on Kore
k
500 releases per year would, for me, be on the other side of what is “reasonable” to expect from a free hosting provider, even if it’s all free and open source on the developer side
➕ 1
h
@Ayfri do you know snapshot releases? But I don’t know if Maven central will limit snapshots too.
a
Yes I do, for each pushes If too limited I'll see to host Minecraft snapshots as SNAPSHOTS and only full round releases as real releases Or even see if self hosting the repo would be not too expensive
b
14,000 files per release. 🤦
Snapshots are meant for internal team sharing primarily only, not for public consumption. These are not currently included in the file counts, but we will be keeping an eye on that as well, as it’s another area right for abuse creative interpretation. 😉
👍 1
We will make adjustments before the deadline, this is all useful feedback. Long term goal is still to work to make many of these tools more efficient for sustainability.
z
That's not how anyone has used snapshots in my experience
Every library I know of that uses snapshots has done so so the community can test pre-release versions
☝🏽 1
☝️ 5
b
In open source land, internal has a wide grey area right.
Distribution from snapshot repo is even more expensive than from Central proper.
z
I guess? When you say internal team I think of coworkers
b
well, i mean intra-team really, but that’s harder to enforce with lots of contributors
and yeah, community testing pre-release is kinda intra
anyway, tangent. Point is, it’s not part of the current limits, but we need to also get that under some form of sane monitoring/control as well.
r
I think the whole idea of snapshots was to save resources 😉
b
I think the whole idea of snapshots was to save resources
care to elaborate?
r
An alternative is to publish full releases more often and keep them forever.
b
oh right, got it
m
In GradleUp, we started publishing SNAPSHOT to a GCP bucket and I like it. Less stress for Maven Central + it gives us full control over the retention, we can advertise that to users.
In most cases, a retention of "1 version" is even probably enough for SNAPSHOTs so most projects out there can get SNAPSHOTs for free using GitHub pages or the likes
And users have to add a SNAPSHOT repo anyway so whether they lookup 
<https://central.sonatype.com/repository/maven-snapshots/>
or 
<http://storage.googleapis.com/gradleup/m2/>
is the same order of added friction.
k
What is the policy / expectation for snapshot releases? My rule of thumb is that I would expect the last ten snapshots to be available, and the rest get dropped after a short grace period.
a
I'm alone developing my project so I don't need SNAPSHOTs for myself, but some users asked me for SNAPSHOTs
m
I'm not even sure the retention policy of maven snapshots is publicized?
I always treated it as "best effort"
c
I created a new thread for SNAPSHOT discussion: https://kotlinlang.slack.com/archives/C8C4JTXR7/p1783345529706389 Let's keep the current thread about the stable MavenCentral repo
👍 1
v
@pdvrieze Our medium-term plan (aka "we are on it right know but it's really hard to estimate") is to have a single properly-compressed artifact per module (instead of $n of targets supported) -- basically https://youtrack.jetbrains.com/issue/KT-87206/Investigate-klib-of-klibs, it should address both file size and number of files restriction. While it's not ready, we'll release public communication when the decision about the current state of klibs and potential exemption for OSS KMP developers is made -- so there won't be a spike of support requests for Sonatype folks and timelines/course of action is clear for everyone. Measures that can help right now -- using the latest gradle-maven-publish-plugin that filters out redundant checksums. If you are using 
maven-publish
-- manual filter of redundant checksums like on [1] will help. For 
javadoc.jar
-- it is currently redundant, but it is not our call and the restriction is up to Sonatype folks to lift
Another thing that is almost sorted out is a potential deprecation of AndroidNative and OS X x86_64 targets -- when the decision is made, I believe we can stop publishing these targets in all KMP libraries
r
@Vsevolod Tolstopyatov [JB] Is there a youtrack ticket for that potential depreciation?
c
Not sure about AndroidNative, but OSX x86_64 targets are deprecated since Kotlin 2.3.20 • kotlinlang.org/docs/whatsnew2220.html#… • kotlinlang.org/docs/native-target-support.html#…
s
gratitude thank you 1
c
@sergey.bogolepov that last one is not publicly visible, or the link is incorrect
s
Oof, my bad :D KT-78660 is a better public-visible one
🙏 1
b
For every publisher weighing in here, I want to nudge you also to take a look at this other thread and let me know your thoughts. https://kotlinlang.slack.com/archives/C8C4JTXR7/p1783711026808809
As an update on progress, we’ve been fielding a lot of inbounds on this and are converging on updates on how the limits will be applied, as well as better defined pricing for the clearly commercial aspects. I’ve been working on the backend analysis and tooling to apply the adjusted limits etc so don’t worry if you haven’t received anything on the request tickets yet, I will process all of them before any actual blocking gets applied.
👍 3
z
Hey Brian, do you know when developers will hear back about exemptions? I haven't seen any change to the published/planned limits and I'm still well over on just a couple projects. Have not heard anything about the exemption request I sent a couple months ago nor has anyone else I've chatted with. Jetbrains is investigating 
.kar
files but that's a long way out at best, and I'm struggling to find what this thread precipitated around the existing usage limits. I know it's a soft cap, but it doesn't feel sustainable to feel like you're just on borrowed time. That also only will help with KMP projects, and not some of the other ones that have come up like gradle plugins. I also have not seen any update on pricing options if users need more. Could we get an update given the emails going out today?
➕ 1
the email today only briefly mentions it and, ironically, the hyperlink isn't actually even clickable
s
For the record, kar's won't address my usecase either (kotlin/jvm with native per-os native compiled helpers and a few optional modules)
b
we published an updated news article last week-ish central.sonatype.org/news/20260716_publisher_pro_update I haven’t processed any of the adjustments yet, the team was rolling out a new implementation based on a bunch of feedback that I needed (new levers to adjust basically)
z
is there a time frame for that? I'm not sure how to balance being asked to prepare for limits but not having anything concrete to do
b
if your stiff is really open source then there’s not much for you to do. If it’s not, then we should connect and figure it out.
h
Do you use the spdx id in the POM file for the open source check?
z
it is really open source, but the messaging right now is a lot of official messaging saying limits are coming vs you in a slack thread saying not to worry 🙂. That will be a particularly tough sell for companies doing open source too I think
j
The current status for us as OSS maintainers is that we've made the request for an exception, but have not received a reply confirming if that will be granted, been told not to expect a reply for a while, reinforced that the limits are still coming, but the limits will probably be tweaked by an unspecified amount.
➕ 1
b
I also feel like the wording in official resources is a bit unclear. At least for me, who is not a native English speaker, it kind of reads as: "Limits are coming. If you are doing OSS work without being paid, no worries! Limits can be increased for you. If a company is backing genuine OSS work, and the limits are too low, you'll need a paid plan"
j
the community has kicked off various efforts to help reduce file size and count when publishing but these changes are likely not sufficient to stay under the current know limits, and changes to KMP probably wont land before the deadline. We've given lots of feedback on how the practical realities of publishing libraries interact with maven central's requirements for javadoc/sources jars, checksums etc, but no changes have been made based on that feedback on maven central's end.
b
@Bernd Prünster @Josh Friend have you seen the update i published above, i think i addressed all of this there. If yes, and it’s still unclear then I can try to improve it.
b
yes, i have written to support after July 16, and read through it. While I have received a response, I still do no know which of the two categories our work falls into (Company or Commercial Publishing, or Community Open Source Publishing)
b
@hfhbd no because many things are open washed. Just because it has an open source license doesn’t mean it’s a free for all any more. That’s what we are trying to define but is hard to implement.
eg sdks for a paid service are really commercial, not open source regardless.
or another case we found a company producing and consuming artifacts in central and no one else uses them… because they are basically commercial internal artifacts and they’ve used us as part of their internal dev pipeline
👍 1
and in other cases, people fake the source to get around the requirement. (i’m building something to crack down on this also)
b
> I still do no know which of the two categories our work falls into (Company or Commercial Publishing, or Community Open Source Publishing) @Brian Fox To clarify: all the work our team is doing (so everything that is published to maven central) is genuine OSS (Apache 2.0-license), and we engage with third parties, receive bug reports, implement features based on request that we don't even need ourselves, … just to make the thing we produce better for everyone, in the hopes to attract more users, to attract more feedback, to make it even better, etc. etc. BUT we are being paid to do so. Does getting paid for the work make it corporate or does acting according to the open source community spirit make it community work?
z
Brian all due respect, several people have said the updates you've given are unclear, and your response has been to just say the updates should make it clear. There is a disconnect here. The only change so far has been slightly raised limits that are not enough, but otherwise it's a lot of writing about gestures broadly bad actors that are not in this thread and very little about how the good actors actually in this thread are to proceed other than “sit tight you're OSS no problem”
I think you'll find people happy to support sonatype when it comes to hosting and what they do for the community, but without clear, written documentation on how to do that it is going to feel increasingly like there could be a rug pull. Some threads I'm in around this are circling back to considering alternatives due to the lack of clarity and changes based on the feedback.
Like, I'm a pretty active OSS contributor/maintainer. I'd love to pay sonatype $5/mo or to support your continued valuable work in the community and never have to think about this again. But instead I am getting emails warning me about impending limits that I just cannot get underneath with current publishing tools
💯 2
b
While I generally agree with @Zac Sweers, I don't want to shift all the blame on Sonatype. I fully understand that bills need to be paid, and I also think I have a good understanding what lead to this situation and I also understand that things need to change. Where I do fully agree is that communication is problematic. Still, probably easier said than done to refine the wording in official resources, to not open up any loopholes, but still: Needs to improve, IMHO. What also does not help is that the defaults of all the bits and pieces involved in producing KMP publications are far from playing together nicely and even farther from being aligned with what would ease much of the burden on maven central. While we've been all-in on KMP for years now, it still feels like you have to actively fight the toolchain to get your publications the way you want them, and you have to jump through even more hoops to add placeholders READMEs referring to the location of hosted Dokka documentation, manually strip checksums, etc. Then there are two disjoint dependency resolution algorithms: one for non-JVM KMP targets, and Gradle's built-in dependency resolver and you can reach situations where they disagree, which can also result in broken publications. But you only realise that they are broken, after you try to consume them from maven central, then you have to republish, which inflates your quota, etc., etc. Bottom line is: I think we are seeing the consequences of no involved party tending to any of the known issues and not engaging with each other for years and that may now culminate in a perfect storm
b
@Zac Sweers my intent is to identify the KMP projects, figure out what the right reasonable levels are and make those adjustments. So there’s likely nothing you need to do here. I’ve said this above a few times, but acknowledge it gets lost.
b
To me it feels like a familiar pattern: • no malicious intent anywhere in sight • we all want a solution • now there are pressing issues • people tend to get pissed All of which means things could have been much easier for everyone if action was taken sooner, when things were calmer. and in the end, nobody is really to blame and we've all been on the same page to begin with, but we've lost sight of that and now we start playing a blame game that nobody can win
b
But importantly I don’t want to just blow the doors wide open because then things like the KMP creep up without an inspection point. That’s why I’ve learned that having some limits still is a useful inspection point
👍 2
but we’ve been trying to sort out and do all the discovery around all the edge cases. We chaned entirely how we were thinking about the limits and the adjacent paid tiers, and that’s what I needed to figure out first. I pushed the limits adjustements out while I dealt with that first. I realize that leaves you hanging, but I’m trying to take this in logical order
s
This is yet again me saying "a project doesn't even need to be KMP to essentially be enough to blow you past quotas with a bi-weekly release cadence". Don't overfit on KMP, because yes it is by far the easiest way to blow past limits, but not the only one at all.
b
understood
i didn’t mean to say that was the only thing I planned to do, simply that it was likely a simple way to put a bunch out of scope in one shot, leaving other things to have a closer look.
👍 1
s
Gotcha, thanks Brian! (btw I sent the OSS exception request today, your mail was a good reminder I needed to)
b
Also. We have changed things around so that we will have an automatic grace period that resets based on some configurable time frame (I don’t have the full numbers worked out yet) but the goal is that non-continuous bursts above the limits don’t do anything but perhaps warn. But month after month of being over limits will create the pressure we desire. So that will reduce a whole bunch of “what if we got it wrong” concerns.
and eliminate a concern for a bunch of vuln releases in a short time frame.
Because I’m really after the long view of back pressure, not trying to interfere with normal ebbs and flows. That’s what we had to go back and do some additional design around
@Bernd Prünster i looked at your stuff. Getting paid isn’t a disqualifier, and this all looks like it falls into the spirit of open source and not commercial/openwashed. Check your usage center and see if new numbers show up. Just experimenting with the new apis and you get to be first 😉
thank you color 1
Ok so I finally had some time to do analysis. @Vsevolod Tolstopyatov [JB] gave me a big list of known KMP projects. I pulled that in, mapped to our namespaces, found obvious file patterns to identify additional KMP projects, and then did telemetry analysis to understand what a P90 for this set would look like. (we already know that KMP tends to be in the top 1% globally, so while the default current limits are +-90% for normal projects work, it doesn’t work for almost all KMP. This is the raw output, just sharing it here for feedback and comparison to what everyone is currently seeing in their usage centers.
Copy code
# KMP Limits Analysis

Generated: 2026-07-29

## Dataset

| | Count |
|---|---|
| Total namespaces | 3027 |
| Known (from seed CSV) | 2324 |
| Discovered (fingerprint) | 703 |
| Unique orgs | 655 |

## Percentile Analysis — 30-day Window

| Metric | P50 | P75 | P90 | P95 | P99 | Global P90 |
|---|---|---|---|---|---|---|
| release_count | 1 | 3 | 11 | 23 | 109 | 7 |
| file_count | 106 | 1512 | 5990 | 15402 | 71339 | 1128 |
| total_release_size (GB) | 0.0 GB | 0.0 GB | 0.3 GB | 1.0 GB | 5.0 GB | 0.1 GB |

## Percentile Analysis — Smoothed (90d ÷ 3)

| Metric | P50 | P75 | P90 | P95 | P99 | Global P90 |
|---|---|---|---|---|---|---|
| release_count | 1 | 4 | 11 | 20 | 95 | 6 |
| file_count | 517 | 1970 | 6906 | 14794 | 80107 | 1172 |
| total_release_size (GB) | 0.0 GB | 0.1 GB | 0.3 GB | 1.0 GB | 4.4 GB | 0.1 GB |

## Proposed KMP SOFT Limits

Based on P90 of the KMP distribution:

| Metric | Window | Proposed Limit |
|---|---|---|
| release_count | 30d | 11 |
| file_count | 30d | 5990 |
| total_release_size | 30d | 0.3 GB |
| release_count | smoothed | 11 |
| file_count | smoothed | 6906 |
| total_release_size | smoothed | 0.3 GB |

Key observations:

- File count is the dominant KMP outlier. KMP P90 (5,990 filesP90 (1,128). Makes sense: one KMP release fans out to 15–20target artifacts.
- Release count is only mildly elevated (11 vs global 7). KMP ore often — they release bigger releases.
- Size stays modest (0.3 GB / 30d at P90). Lots of small .klib / metadata files, not big binaries.                                              - Long tail is heavy — P99 file count is 71,339 (30d) and ~100 spaces are doing 10× the P90 volume.
r
The proposed file count still feels low and would allow me to only publish a single release per month.
b
what’s your namespace?
z
same for me
r
software.ralf
is my namespace
z
also on behalf of @seb reminding that this isn't unique to KMP 🙂
❤️ 1
🙏 2
r
It counts as two releases, because the Gradle plugin for this project is published separately.
➕ 1
a
Do they reply to quota augmentations requests ? I sent an email a month ago but still don't have any reply
b
@Ayfri they = me. I posted a news update about this. central.sonatype.org/news/20260716_publisher_pro_update
I’m working on the obvious KMP problem first to understand how to make a bulk adjustment to hopefully result a giant chunk all at once. Then we will work through the rest of the list.
c
The "key observations" in your previous message match my experience (
dev.opensavvy
). With regards to the proposed P90-based limits, where 'ok' means 'this limit would not trigger in our usage':
Copy code
| Metric | Window | Proposed Limit |
|---|---|---|
| release_count | 30d | 11 |                          ok (4–5)
| file_count | 30d | 5990 |                           no, we can reach 20k
| total_release_size | 30d | 0.3 GB |                 ok but we're near it
| release_count | smoothed | 11 |                     ok
| file_count | smoothed | 6906 |                      no, we can reach 30k
| total_release_size | smoothed | 0.3 GB |            ok but we're near it
In our case, the release of 
.kar
(whenever it happens) will have a massive impact and put us well below the limits.
b
thanks for that correlation
j
here's the analysis of what our usage of the proposed limits would be based on the last year of releases i pulled from our namespaces. misk/wisp is a server framework that has regular (multiple weekly) releases, so I made a chart with and without that project included.
what stands out to me when i think about this is that measuring file count/size and release count doesn't feel like it has any strong correlation with openwashing or commercial activity. I suppose something like the aws sdk is an obvious counterpoint to this, but it seems to have a similar publishing cadence to something like misk or the android jetpack compose libs for example. Applying the limit to an entire account seems to blunt and perhaps per-namespace or per-maven-group-id would be better? I don't imagine that splitting a commercial sdk across maven groups would be a workable way to try to circumvent limits.
b
I’ve had similar conclusions yes.
b
Applying the limit to an entire account seems to blunt and perhaps per-namespace or per-maven-group-id would be better?
Sounds promising to me, as we have a single namespace, but a couple of groups in that namespace. Related? How common is it even to have multiple namespaces that are governed by one entity?
b
That’s why in my update, I said we were moving to separate the two a bit more, but the limits are really having an impact. We were talking to one (also commercial vendor) who cut the size of their stuff by 200x just by looking and taking out waste they didn’t need.
🎉 1
@Bernd Prünster it’s pretty common actually, but as we only recently introduced the new org level to bind namespaces, the data is messy
👍 1
eg a bunch aren’t linked to an org but they probably actually should be
we see lots of x.y.foo and x.y.bar type of stuff
v
for KMP, our plan is to make an experimental (e.g. not enabled by default, but heavily endorsed) release of .kar in 2.5.0. TL;DR: all packed into a single artifact/coordinate, so number of files on par with a regular publication, the same for the size. It is scheduled for ~first two weeks of December, and the first Beta is for end of September. Before that, we can only rely on exemption rules (+we are officially deprecating some T3 targets, +maven publish plugin update with checksums should help)
🚀 2
b
That’s a great optimization, thanks for sharing
h
As a consumer, I would need to update KGP 2.5.0 to consume libraries published with the new model, wouldn't I?
v
Correct; we are also shipping hotfix to 2.4.20 that will properly acknowledge existence of .kars: https://youtrack.jetbrains.com/projects/KT/issues/KT-87911
b
Interesting data. I analyzed all namespaces published in the last 12 months. KMP namespaces make up about 22% of those spaces but 93% of the total artifacts published in that time.
😯 1
@Paul Horton fyi.
o
TestBalloon (namespace 
<http://infix.de|infix.de>
) was in P99 on file count, because each release actually came as a set of 8 releases for different Kotlin compiler versions. The compiler plugin infra has now been restructured, leading to one integrated release instead of 8, with a file count down from 52470 to 2332. The proposed P90 limits would now allow two releases per month.
🎉 1
k
Overall it looks like a much overdue nudge for the Kotlin ecosystem in general, to have a critical look at all the steps across multiple toolchains at what is being produced and what is being distributed.
👍 2
o
I'd not over-generalize the TestBalloon case. That project was providing an extreme range of backward compatibility with the latest fix/feature set. And making a single compiler plugin artifact auto-adapt to a range of Kotlin compiler versions is a sizable effort and not for the faint of heart. That said, I'd expect the new Kotlin archive publication format (.kar) to become the most decisive contribution towards shrinking the KMP library footprint for everyone.
k
Not just this case, but quite a few others since this thing was started two months ago
121 Views
Open in Slack
PreviousNext
Powered by