https://kotlinlang.org logo
Join Slack
Powered by
Hi everyone. Can anyone here help me understand wh...
# compiler
b
Hi everyone. Can anyone here help me understand why the kotlin-compiler-embedded component downloads are consuming a full 20% of the ENTIRE Maven Central bandwidth and what blows up if those get rate limited? It seems like a runtime dependency not a build time dependency which is not what we would normally consider “in scope” for Central… certainly not at 20% of the entire bill.
p
That's the kotlin compiler itself. So, essentially every kotlin build in the world would need to download it.
h
But you normally download it once and cache it via Gradle/Maven dependency cache.
b
Empirically that isn’t happening with ips downloading it 10,000 a week.
something like 50 of the top 75 components by bandwidth are all these comilers.
(maybe more, it’s pages and pages of them)
p
Do you mean different versions of this artifacts?
b
yeah looks like it. But point being it’s not just one, it’s like all of them
h
But if you build a Kotlin application/library with a clean CI environment, I would expect such numbers.
p
I would expect it to be on-par with central.sonatype.com/artifact/org.jetbrains.kotlin/kotlin-stdlib on number of downloads (but probably the artifact itself is much bigger). Can you check it?
v
that's something very suspicious, it shouldn't be that big. Is it possible to look at some detailed sample of the downloads? Anything that can pinpoint the issue: UAs, version distributions etc. > I would expect it to be on-par with stdlib It should be on-par count-wise, but size-wise 
k-c-e
is like 20 times bigger (50 mb?). So maybe downloads-wise it is a reasonable number, but artifact-size-wise it contributes way too much
p
Sure. I suggested it more as a sanity check of numbers. If nummer of downloads differs significantly that can be a signal of something more broken than we thinking.
h
Gradle also uses the Kotlin compiler for KTS scripts, so using Gradle 9.7.1 to built a Kotlin 2.4.20 application downloads the Kotlin compiler twice.
b
why twice?
stdlib is 165k vs compiler-embeddable at 37k in the same timeframe.
The issue is really the size and how it doesn’t seem to be cached or is downloaded on every build or something.
h
Gradle has two DSLs: Groovy and Kotlin. And the Kotlin DSL requires a Kotlin compiler, often a different version than the user uses to compile its project. But I guess it's mostly caused by uncached CI builds.
a
I would not that even pure Java projects at this point get kotlin-stdlib as it is often a transitive dependency of many libraries in the JVM ecosystem
My speculation is that k-c-e is largely redownloaded by GitHub actions ephemeral runners
➕ 2
I know what Gradle made a special deal with GitHub to mirror/intercept downloads to Gradle-*-bin.zip as there was so much traffic from GitHub actions
we might want to do a similar set up here
p
Official gradle actions with caching support should help to avoid this, but I can imagine people massively not using them, and just running grade command with default execute shell command step.
h
JFTR: there is an official GutHub action for Gradle to setup the cache, but yeah, you have to use it. https://github.com/gradle/actions
b
Similarly, we’ve worked with React to do redirects for the large react native tarballs. They setup a separate pull through mirror and are moving that traffic. We are doing redirects for that targeted traffic now. That could be something to consider here. It makes the publishing and archiving easy, it’s all still in Central, but for these unsually large / different artifacts they get hosted elsewhere.
My overaching goal is not to “cost shift” but rather look for ways to reduce the impact and if there isn’t one, then we would ask for some way to offset the costs (eg a commercial agreement). That is the last resort for cases like this.
@Pavel Kunyavskiy [JB] who from Jetbrains can we have direct conversations with about this? We have been in touch with others about the publishing limits and things like TeamCity traffic, but I suspect this is a different part of the org.
h
You could also publish the compiler.jar as a GitHub release asset (additional to the existing zip file). According to the GitHub docs, there is no outgoing limit.
IMHO: if CI builds using GitHub Actions are the main consumers, GitHub Actions (users) should use artifacts published at GitHub instead, eg via release assets or via GitHub Packages to keep the traffic in their network.
b
That could make sense for these relatively small number of large files. I could even do a redirect like react if needed.
p
I'm a good contact point for technical questions. About any agreements probably @Vsevolod Tolstopyatov [JB] is good starting point.
v
The discussion went into a private space, for the further public discussion and improvements, here is https://youtrack.jetbrains.com/issue/KT-89442/Gradual-reduction-of-kotlin-compiler-embeddable-distribution-size
b
That’s a great start @Vsevolod Tolstopyatov [JB]. I’ll get a bit more details per your questions
a
More data would be nice. Can we get a breakdown by versions? If it spikes on versions that Gradle uses that's a big hint. Maybe Gradle could specially cache it in the global Gradle cache. Are the downloads coming from Gradle or Maven? We could get an indication by seeing how many times the pom vs module files are downloaded.
b
Copy code
The kotlin compiler embedded family moves as a unit — all three are downloaded together in Gradle Kotlin DSL builds:

┌──────────────────────────────────────┬───────────────┬───────────────┬────────────┐
│               Artifact               │ JAR downloads │ POM downloads │ Unique IPs │
├──────────────────────────────────────┼───────────────┼───────────────┼────────────┤
│ kotlin-compiler-embeddable           │ 46.4M         │ 44.5M         │ 3.7M       │
├──────────────────────────────────────┼───────────────┼───────────────┼────────────┤
│ kotlin-daemon-embeddable             │ 45.6M         │ 43.9M         │ 3.7M       │
├──────────────────────────────────────┼───────────────┼───────────────┼────────────┤
│ kotlin-scripting-compiler-embeddable │ 37.0M         │ 35.5M         │ 3.5M       │
├──────────────────────────────────────┼───────────────┼───────────────┼────────────┤
│ kotlin-compiler (fat jar)            │ 1.6M          │ 1.6M          │ 286K       │
├──────────────────────────────────────┼───────────────┼───────────────┼────────────┤
│ Total (jar+pom across all)           │ ~255M         │               │            │
└──────────────────────────────────────┴───────────────┴───────────────┴────────────┘

~8.5M downloads/day for just this component group.
thank you color 1
Copy code
User Agent Breakdown (kotlin-compiler-embeddable, 30 days)

┌──────────────────────────┬───────────┬───────┬────────────┬─────────────┐
│        Proxy/Tool        │ Downloads │   %   │ Unique IPs │ UA variants │
├──────────────────────────┼───────────┼───────┼────────────┼─────────────┤
│ Gradle                   │ 85.0M     │ 93.5% │ 3.68M      │ 617,890     │
├──────────────────────────┼───────────┼───────┼────────────┼─────────────┤
│ Coursier (Scala)         │ 3.0M      │ 3.3%  │ 39.7K      │ 2           │
├──────────────────────────┼───────────┼───────┼────────────┼─────────────┤
│ Apache-Maven             │ 2.5M      │ 2.7%  │ 153.9K     │ 20,891      │
├──────────────────────────┼───────────┼───────┼────────────┼─────────────┤
│ Wget                     │ 59K       │ 0.07% │ 7.4K       │ 12          │
├──────────────────────────┼───────────┼───────┼────────────┼─────────────┤
│ curl                     │ 56K       │ 0.06% │ 12.7K      │ 87          │
├──────────────────────────┼───────────┼───────┼────────────┼─────────────┤
│ KotlinToolchain          │ 16.6K     │ 0.02% │ 4.7K       │ 8           │
├──────────────────────────┼───────────┼───────┼────────────┼─────────────┤
│ Nexus (proxied)          │ 27.8K     │ 0.03% │ 4.7K       │ —           │
├──────────────────────────┼───────────┼───────┼────────────┼─────────────┤
│ Artifactory (proxied)    │ 23.6K     │ 0.03% │ 2.1K       │ —           │
├──────────────────────────┼───────────┼───────┼────────────┼─────────────┤
│ DCE-Services (Checkmarx) │ 9.3K      │ 0.01% │ 67         │ —           │
└──────────────────────────┴───────────┴───────┴────────────┴─────────────┘

Gradle owns this completely — 93.5% from 3.68M unique IPs. The 617,890 distinct UA variants is not chaos — it's Gradle's convention of embedding the Gradle version + JVM version + OS in every UA string, so every CI runner combination produces a unique UA. This is the Kotlin DSL build path: any build.gradle.kts file causes Gradle to fetch kotlin-compiler-embeddable to compile the DSL scripts at build time.
thank you color 1
Copy code
Cloud Provider Breakdown (kotlin-compiler-embeddable JAR, 30 days)

┌───────────────────────────────┬───────────────┬───────┬────────────┐
│           Provider            │ JAR Downloads │   %   │ Unique IPs │
├───────────────────────────────┼───────────────┼───────┼────────────┤
│ Azure (ASN 8075 / Microsoft)  │ 19.8M         │ 42.6% │ 70.2K      │
├───────────────────────────────┼───────────────┼───────┼────────────┤
│ Other (residential, ISP, VPS) │ 15.7M         │ 33.9% │ 2.26M      │
├───────────────────────────────┼───────────────┼───────┼────────────┤
│ AWS (ASN 14618, 16509)        │ 8.1M          │ 17.5% │ 1.17M      │
├───────────────────────────────┼───────────────┼───────┼────────────┤
│ GCP (ASN 396982 / Google)     │ 2.8M          │ 6.0%  │ 208.7K     │
└───────────────────────────────┴───────────────┴───────┴────────────┘

Total cloud: 66.1%. Azure at 42.6% reflects GitHub Actions + Azure DevOps dominance for Kotlin/Android CI. The "Other" bucket at 33.9% but with 2.26M IPs is the long tail of individual developers, residential networks, and small VPS providers — highly distributed, low-volume per IP.
thank you color 1
It you are able to put the embeddable jars in github releases, then I think as an interim solution, we can do a rewrite rule to direct the github actions traffic back to that source. According to above, that would cut about 40% of the traffic. This is not traffic or cost shifting imo as it keeps it inside the same network, it’s actually more efficient.
➕ 1
a
Really cool to see the stats Brian, thank you
Uploading kotlin-compiler-embeddable would be possible - but given that it participates in the Gradle dependency resolution and has transitive dependencies it'd need some special handling to make sure Gradle understands how to resolve it. Gradle can support resolving artifacts uploaded to GitHub Assets, so they'll be treated as 'regular' dependencies, but it's a bit clunky. Fixing this issue would help https://github.com/gradle/gradle/issues/33674.
a
I think the suggestion is to have an HTTP redirect to github assets to it would entirely transparent to gradle
nice spin 1
h
Yes
a
Like 308
or 307
b
@Vsevolod Tolstopyatov [JB] these are the top 5 over the last 30 days:
Copy code
/maven2/org/jetbrains/kotlin/kotlin-compiler-embeddable/2.0.21/kotlin-compiler-embeddable-2.0.21.jar
280.58 TB
/maven2/org/jetbrains/kotlin/kotlin-compiler-embeddable/2.4.10/kotlin-compiler-embeddable-2.4.10.jar
148.94 TB
/maven2/org/jetbrains/kotlin/kotlin-compiler-embeddable/2.4.0/kotlin-compiler-embeddable-2.4.0.jar
146.75 TB
/maven2/org/jetbrains/kotlin/kotlin-compiler-embeddable/2.2.21/kotlin-compiler-embeddable-2.2.21.jar
136.67 TB
/maven2/org/jetbrains/kotlin/kotlin-compiler-embeddable/2.2.20/kotlin-compiler-embeddable-2.2.20.jar
Lets pick a few of these to put in your Github registry and we can try some limited rewrites to see how that goes. Or if you like, we can start with one further down the list, but I would like to make a move on it this week. The work in your other ticket to make everything smaller is still really important, but this has immediate impact on costs and there’s no real reason to wait.
v
we are on it, I'll keep you posted
m
@Brian Fox I've uploaded artifact for 2.4.10, can you please check? GitHub already shows SHA-256 automatically, but if you need Maven-style checksum files too, please let me know which format you expect. github.com/JetBrains/kotlin/…/v2.4.10
thank you color 1
b
cool. let me work on this, a redirect for just the jar makes the most sense
(on a plane with not great wifi so this may take a while….)
@Vsevolod Tolstopyatov [JB] @Margarita Bobova this is in place now currently just for the embedded jar. I’ll make it scoped to the MS ASN shortly but left it more open for testing. Once we are comfortable tools don’t do anything unexpected with the rewrite, I’ll look at expanding it to the other zips etc in there, then we can look at more versions.
v
Sure, let's see if it works and then we can also expand uploaded set of artifacts. Mixed news from my side -- your downloading stats perfectly correlate with compiler versions specific Gradle versions are using for their scripting, and the distribution in bandwidth roughly matches our analytics of Gradle versions in the wild. I am yet to figure whether this is for Kotlin projects only, or maybe an unconditional download even for Java+Groovy projects etc.
b
I have some contacts at Gradle if we want to loop them in.
v
I am on my way to Gradle Summit right now, so hopefully we'll be able to discuss it there offline with Gradle folks 🙂
h
Like I said, the Gradle Kotlin DSL also needs the Kotlin compiler so it causes additional downloads
b
We’ve had to go through some similar changes in Gradle as well
v
@hfhbd sure. But it can be the case that it's downloaded even for builds without Kotlin DSL, and it can be the case that ~6 "embeddable" artifacts are downloaded for a single build (compiler, scripting, daemon etc.). We'll pinpoint the actual issue and then it can be addressed
b
Oh gosh. I didn’t understand what you meant before, but I get it now. So people are writing Gradle builds in Kotlin and that grabs the compiler every time Gradle starts? 🤦
👀 1
h
Yes
Well, it is cached
b
allegedly 😉
v
because the case "Gradle downloads a single k-c-e.jar only for .kts builds" is drastically different from "Gradle downloads 6 embeddable jars even for pure Groovy+Java projects"
b
and is the version tied to the gradle version as in could it be included in gradle or the plugin directly?
h
Yeah, but not every start. But if the runner, eg a clean CI runner, does not contain it, it needs to download it.
b
yeah. The size of the files is ultimately what is different here. It’s an exponential issue than a few inefficient small jars
compounded by it’s really hard still for Gradle users to configure a repository. I’ve been going back and forth with them on that for a while.
not a GHA issue, but more general
h
I did some tests by using the GitHub egress firewall runner and the Kotlin compiler embedded jar is only downloaded when you apply the kotlin-dsl plugin, e.g. when writing a custom local plugin. I have no idea why it is not downloaded/needed for build.gradle.kts files without writing local plugins. But writing local conventions plugins is a best practice. You can see the logs of each different test here: https://github.com/hfhbd/foo/actions
b
@Vsevolod Tolstopyatov [JB] @Margarita Bobova the prebuilt components could be useful to redirect, however I noticed the file names don’t match. Can we fix them? Maven Central (…/maven2/org/jetbrains/kotlin/kotlin-native-prebuilt/2.4.10/) — Maven’s mandatory layout: artifactId-version-classifier.ext kotlin-native-prebuilt-2.4.10-linux-x86_64.tar.gz kotlin-native-prebuilt-2.4.10-macos-aarch64.tar.gz kotlin-native-prebuilt-2.4.10-macos-x86_64.tar.gz kotlin-native-prebuilt-2.4.10-windows-x86_64.zip GitHub Releases (github.com/JetBrains/kotlin/…/v2.4.10) — platform before version: kotlin-native-prebuilt-linux-x86_64-2.4.10.tar.gz kotlin-native-prebuilt-macos-aarch64-2.4.10.tar.gz kotlin-native-prebuilt-macos-x86_64-2.4.10.tar.gz kotlin-native-prebuilt-windows-x86_64-2.4.10.zip
👀 1
we can rewrite but the extra complexity might not be worth the tradeoff. Also someone may have an implicite expectation on the maven format given from their pov it came from a maven url that was redirected
m
@Brian Fox I've uploaded K/N archives with MC-like names. Please let me know if you need separate checksum files to be uploaded too.
b
no checksums. Arguably it’s a good cross check to have them validate against what Central has anyway.
I’ll expand the rules a bit and see how that goes and then we can look at adding other versions.
this had a meaningful drop in the bandwidth yesterday so as long as people’s builds aren’t breaking, I think this is looking like success
kodee happy 1
v
amazing news! Would you prefer other artifacts to be on GH releases as well?
b
@Vsevolod Tolstopyatov [JB] it looks like doing 2.0+ would be good. The next highest is 2.0.21. Ideally i could have a more generic rule that sends anything for 2.0 with these tar.gz back to GH so we don’t have to cherry pick.
m
✔️ 2.0.21 ✔️ 2.4.0 ✔️ 2.4.10 ✔️ 2.2.21 ✔️ 2.2.20 ✔️ 2.4.20 Please let me know if there's anything else you need.kodee welcoming
v
@Brian Fox the generic rule would be nice. Let's check with the versions Margarita uploaded (thanks a lot! 🙇), and if it works, we'll ensure that GH artifacts are part of our regular release process and have it handled for good with the blanket rule
(we'll keep reducing the size of the artifact anyway, corresponding projects are being implemented, now they rather have more explicit target about the binary size)
o
Just looking from the sidelines, but this level of responsiveness and cooperation is so rare in our industry that it's a pleasure to observe it here. A big thank you to everyone involved! ❤️
thank you color 1
kodee loving 1
💯 1
b
I agree. Jetbrains have been really helpful getting this to a reasonable outcome. Not only in this thread, but also in the recent improvements to reduce the size of KMP releases generally. Sadly it is rare.
➕ 1
The updated rewrite rules for the additional versions is now in place. If anyone sees or hears anything, please @ me so I can see what’s going on. My positive and negative tests show the rewrites are working correctly and only for the versions we expect.
v
@Brian Fox could you please verify whether the bandwidth problem is gone? Should we start making GH releases part of our release processes?
b
@Vsevolod Tolstopyatov [JB] it is a lot better, today will be the first full workday with good data after the rewrite rules so that will help me assess where we are. I can say that yes for these large artifacts, doing it on GH would be a good standard. I noticed today some 1.9 versions are popping to the top of the list, this time seemingly from Coursier but I can dig in a bit more and see where the hot spots continue to be.
thank you color 1
@Vsevolod Tolstopyatov [JB] @Margarita Bobova here’s the full list of versions to complete the rules (that have significant traffic)
👀 1
2.3.21 2.3.20 2.1.20 2.1.0 2.2.10 2.2.0 2.3.0 2.0.20 2.0.0 2.3.10 2.1.21 2.1.10 2.0.10 And the top 1.9.x versions by bandwidth 1.9.22 (2,580 GB/day) 1.9.24 (2,484 GB/day) 1.9.25 (1,899 GB/day) 1.9.23 (1,633 GB/day) 1.9.10 (1,515 GB/day) 1.9.0 (1,223 GB/day) 1.9.20 (1,118 GB/day) 1.9.21 (400 GB/day)
The gradle-plugin itself generates as much bandwidth as the compiler, likely because it’s the component pulling all of these in. If we can get that included as well on GHR for now it would double the savings
m
Hi, will do! Can you please mention which artifacts do you need, except kotlin-compiler-embeddable and kotlin-native-prebuilt* artifacts? I can upload kotlin-gradle-plugin as well, and I saw kotlin-daemon-embeddable and kotlin-scripting-compiler-embeddable were mentioned above too, do we need to upload them?
b
checking
That’s correct @Margarita Bobova org.jetbrains.kotlin:{kotlin-compiler-embeddable,kotlin-gradle-plugin,kotlin-native-prebuilt}
👍 1
I don’t love this as a long term solution and I’m thinking through some better alternatives, but as a short term for large files coming from GHA it’s really helpful. Longer term we can consider multiple things.
👍 1
m
Done: • 2.3.21 • 2.3.20 • 2.1.20 • 2.1.0 • 2.2.10 • 2.2.0 • 2.3.0 • 2.0.20 • 2.0.0 • 2.3.10 • 2.1.21 • 2.1.10 • 2.0.10 Will upload 1.9.x a bit later👀
b
rules updated. The gradle plugin doesn’t exist on the original versions, but i was able to make it work for just the ones where it is already there ” 2.0.21, 2.2.20, 2.2.21, 2.4.0, 2.4.10 and 2.4.20.”
m
I've uploaded gradle-plugin for 2.0.21, 2.2.20, 2.2.21, 2.4.0, 2.4.10 and 2.4.20. Uploaded all 6 artifacts for 1.9.21, 1.9.20, 1.9.22 , 1.9.24, 1.9.25, 1.9.23 Uploaded Kotlin artifacts for 1.9.10 and 1.9.0. But for these releases, the K/N artifacts were published on the JetBrains download server, not on Maven Central, these artifacts already uploaded to GH release pages.
3 Views
Open in Slack
PreviousNext
Powered by