https://kotlinlang.org
Join Slack
Are there any ongoing activities towards build optimizations? Our internal library has become quite ...
# multiplatform
p
Are there any ongoing activities towards build optimizations? Our internal library has become quite big and is heavily modularized. However especially the sync time is extremely long (about 5-8 minutes) and that is even with multiple optimizations applied that completely disable all native targets through gradle properties. So this is measured on a pure jvm-android project.
➕ 5
m
are you using buildSrc or compositeBuilds ?
p
Composite builds
m
Ohh okay then to me composite builds makes difference but you already have it so no idea, following the thread now on 🙂
p
The only thing making it fast is disabling hmpp
👀 1
l
How many modules do you have?
p
30ish
l
@Sebastian Sellmair [JB] Do you have any internal benchmarks/tests with that many multiplatform modules? If not, I think it'd be helpful to have one, to set our expectations and maybe provide a starting point for optimizations blob thinking fast
p
Iirc it's about 40k loc
The major problem is the ide. Doing things on command line takes time but it's not these minutes of waiting. We've stopped creating new modules due to large sync times.
s
5-8 minutes, when just creating a new module? 😮, with just jvm targets syncing? As described here, this is not acceptable at all to me. Are there ways to get access to that internal library? Is there a good OS project that also suffers in a similar manner?
@louiscad, thanks for pinging me here!
🙂 1
p
@Sebastian Sellmair [JB] I can't share it unfortunately. I just measured it and it's no longer as bad as it used to be. I added a new empty module and it only takes 1:45 min to sync and then an additional 45 seconds to index
When I flip hmpp to: kotlin.native.enableDependencyPropagation=true kotlin.mpp.enableGranularSourceSetsMetadata=false Its 45 seconds sync + 3 seconds indexing
On the command line it doesn't seem to make any difference: ./gradlew jvmJar --rerun-tasks --no-build-cache 55 seconds without hmpp 56 with hmpp
with just jvm targets syncing?
jvm+android+commonJavaMain being the parent of both
And it's 48 modules, not 30. And 32k loc, not 40k
s
Okay, thanks for the info! Do you maybe roughly know what you’re waiting during syncing? What is intellij telling you it is doing? “Build gradle model”? Do you have any idea what it starts to re-index syncing?
p
Yep mostly building model
s
Okay, so not running any Gradle tasks pre-sync, right? I think we can already work with that.
p
What do you mean by
Okay, so not running any Gradle tasks pre-sync, right?
s
During IDE sync, Gradle tasks might. be running, like custom artifact transformations, commonizer, or just preparation of the buildscript model.
p
Nope doesn't look like.
Copy code
Type-safe dependency accessors is an incubating feature.
Type-safe project accessors is an incubating feature.
> Task :plugins:extractPrecompiledScriptPluginPlugins UP-TO-DATE
> Task :plugins:generateExternalPluginSpecBuilders UP-TO-DATE
> Task :plugins:compilePluginsBlocks UP-TO-DATE
> Task :plugins:generatePrecompiledScriptPluginAccessors UP-TO-DATE
> Task :plugins:configurePrecompiledScriptDependenciesResolver
> Task :plugins:generateScriptPluginAdapters UP-TO-DATE
> Task :plugins:compileKotlin UP-TO-DATE
> Task :plugins:compileJava NO-SOURCE
> Task :plugins:pluginDescriptors UP-TO-DATE
> Task :plugins:processResources UP-TO-DATE
> Task :plugins:classes UP-TO-DATE
> Task :plugins:inspectClassesForKotlinIC UP-TO-DATE
> Task :plugins:jar UP-TO-DATE

> Configure project :changesIndicator
Unable to detect AGP versions for included builds. All projects in the build should use the same AGP version. Class name for the included build object: org.gradle.composite.internal.DefaultIncludedBuild$IncludedBuildImpl_Decorated.
Warning: Kotlin language settings property 'experimentalAnnotationsInUse' is deprecated and will be removed in next major releases. Please, use 'optInAnnotationsInUse' instead.
And it takes a long time indexing things
s
Thanks a lot! I will hope being able to either tackle import performance myself in some reasonable time, or will nudge others 👀 I’ll keep you updated!
🙂 1
l
If you struggle to get a reproducer, keep in mind my open source project Splitties is one regarding slow IDE indexing.
p
Also with Kotlin poet you can easily set up a n-modules x-loc project yourself for Benchmarking
s
Is poet a mpp library? 🤔
l
Not itself AFAIK, but it can write Kotlin code for any targets. It's used in SqlDelight for example.
s
Hmm. I think I at least found one obvious way to reduce the number of libraries to index drastically, so that in theory adding a new module would not even require any indexing 👀
🤩 3
p
With poet you could create a script that creates a project with a defined number of modules each containing a defined number of classes, functions, whatnot. That would be my first approach for Benchmarking with multiple paramters
s
Ahhh! Got what you meant 👌
l
A great motive to try the first EAP that would include this optimisation 👀
s
I do not see a reason why adding a new module should take any longer than 10 seconds to sync.
p
One obvious way
Can you describe what you have found? In curious
s
l
I think this would deserve a prominent before/after comparison in a blog post regarding the indexing speedup.
Oooop, looks like I've been overhyped too fast 😅
s
No, I would not propose to speedup indexing (though there seems to be potential improvements) But I would just try to avoid indexing here 👀
l
Well, technically, it's an infinite reduction in indexing time.
p
Four our project indexing is only a part of the story
s
Where does it also hang?
Any input will be appreciated!
l
Splitties usually also takes one minute for a Gradle sync, even without adding a module
s
Long time between those Gradle tasks ands before indexing?
p
The building module thing
s
Okay, got it!
p
How long does it take for splitties?
l
How long does it take for splitties?
What exactly?
p
Sync+index
s
Here is technical detail on how to avoid indexing after adding a new project: Right now, metadata (Kotlin headers) are shipped in this -all.jar artifact. This artifact is a composite artifact that includes the metadata compilation of multiple source sets of a Library (and a little more). Right now, a transformation “GranularMetadataTransformation” is running on those artifacts to extract the corresponding klibs from that artifact. Right now, those are scoped by Gradle project and will create new transformation artifacts per Gradle project. Those can be stored globally and then the same library can be used for analysis in the IDE. If it is already indexed, no reindexing will be required after adding a new module. Sorry for the very short explanation. If you folks are interested, I can give you a more in depth technical presentation virtual meeting ? But for now: It’s weekend!!! 🎉 🥳
👀 1
p
I have vacation, feel free to start you presentation now
I'm in the garden with my kids, they would love it too!
l
Up to 10 minutes for almost 50 modules
p
Ouch
s
Unfortunately, I have an appointment in a Bar right now 🙃 Feel free to join in Rosenheim, Nenas 😅
😄 2
I will very happily talk about indexing performance there
p
Really? We are going to Rosenheim on monday
s
Not really??? Rosenheim, Germany, Bavaria?
Wanna meet up then?
p
Yeah sure :) I'll drop you a pm
Is there an issue that's tracking this?
s
Not yet! I have implemented the optimisation in a PR for KPM, but not the current multiplatform implementation. How heavy are you suffering? Would you mind jumping into a call with me, to evaluate if i should spend resources to also implement this in the current multiplatform model for 1.7.0?
p
Suffering a lot 😉 I’d be available right now
s
Oh, I see! Then would you mind opening intellij, preparing what you can show me and then drop me another message, then we can jump into a call 👍 👍
p
Yep, ide is open
s
Sending link to join in a minute
l
@Sebastian Sellmair [JB] I see I participated in this old thread, should I also join?
s
👍 Yes!
p
s
@Paul Woitaschek: Are you using cinterops in your project?
p
Nope
s
Do you happen to have kotlin.git setup so that you could compile the compiler and kotlin gradle plugin=?
Basically just clone kotlin.git and hit ./gradlew install
p
Yes I have git and java 😁
s
🦜 Then do it 😁 Checkout rra/sellmair/KT-48135-deduplicate-extracted-metadata-klibs Hit ./gradlew install And then in your project use kotlin version 1.7.255-SNAPSHOT and see if you can spot differences!
I think one is still required to setup some environment variables (according to our readme). I scheduled a dev build also, so if you have challenges building kotlin here, the dev build should be available in an hour, when all goes according to plan.
p
Already on the playground, I'll report tomorrow and let you know
s
I really hope, this will help us!!! Properly implementing this would require roughly one working day, which I think is absolutely justified if you can notice a difference 🤞
p
How do I profile this?
s
This is how you would profile the idea part of syncing Just start the profiling, hit the sync button, stop and send me the file ☺️
For profiling the Gradle process during sync, probably async profiler might do: https://github.com/jvm-profiling-tools/async-profiler
p
It fails with a tons of issues in the form of:
Copy code
e: /Users/ph1b/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-compiler-embeddable/1.7.255-SNAPSHOT/56df3d2fef65431fcaf99063e5ca2e44c9a786a1/kotlin-compiler-embeddable-1.7.255-SNAPSHOT.jar!/META-INF/wasm.ir.kotlin_module: Module was compiled with an incompatible version of Kotlin. The binary version of its metadata is 1.7.0, expected version is 1.5.1.

> Task :core:compileKotlin FAILED
s
You updated all kotlin dependencies accordingly to 1.7.255-SNAPSHOT? If so,. let me prepare a branch based from 1.6.20
p
Yep, it’s all defined in the version catalog. Maybe it’s related to ksp, that I cannot easily update
s
I’ll send another branch later 👍
p
Building myself didn’t work on an m1
> Could not read ’https://api.adoptopenjdk.net/v3/binary/latest/8/ga/mac/aarch64/jdk/hotspot/normal/adoptopenjdk' as it does not exist.
s
Can you quickly check if 1.6.20-RC works for your project?
p
Yes it works
s
I pushed a branch for 1.6.20: sellmair/1.6.20/yazio/kt-48135 
./gradlew install -PdeployVersion=1.6.20-yazio
And then change kotlin version to 1.6.20-yazio ☺️ 👍 Hehe, note: I did not test that branch. Also my machine is still compiling it, will drop another message if I push into that branch again. 😬
p
I installed jdk 8 now but now I think it needs even 6 🙈 > No compatible toolchains found for request filter: {languageVersion=6, vendor=any, implementation=vendor-specific} (auto-detect true, auto-download true)
s
Need to fix merge issues anyways 😬 Will ping you once the branch is ready again. 👀
p
Can you push a CI branch later?
s
Yes, I think so 👍
However, CI build will take probably two hours 😕
p
Tomorrow is another day 🙂
l
Is it so long because the CI build also runs all tests?
s
@Paul Woitaschek: Maybe this gradle property helps? kotlin.build.isObsoleteJdkOverrideEnabled=true
dev build will be available roughly 17:30 The reason is that this dev builds also build a full native distribution, which takes quite some time
But the property should work on an arm mac 👍
CI is currently pretty busy, sorry folks! Dev builds will take until tomorrow 😕
p
So the sync feels kind of the same - the indexing feels way faster. I can’t profile it as these options don’t exist in AS: And I can’t use Intellij due to it being incompatible with agp 7.1
Adding a new module and syncing still takes about 5 minutes
l
@Paul Woitaschek it's part of a plugin you can install, it's from JetBrains, let me find the name when I have my IDE in front of me.
s
Okay, I asked for a new channel for us #multiplatform-idea-performance @louiscad: Can we update splitties to 1.6.20 in one branch?
l
@Sebastian Sellmair [JB] Yup, I'll start updating to 1.6.10 first, and we can do it right after.
m
Hi 👋 I’m also facing the same issue and would love to help in any way possible. For me majority time is spent in “Build model…” step around 70-80% of total sync time. The project is using Kotlint 1.6.10 and has around 50 Kotlin multiplatform modules and 25 Android only modules. I should also mention the use of Cocoapods plugin in 2-3 modules. I can try the steps mentioned above, i.e. updating to 1.6.20 and using a dev build and report back with the updates. I’m hopeful that together we’ll be able to solve this issue and improve developer experience simple smile
p
@Mayank Invited you
m
Thanks @Paul Woitaschek
5 Views
Open in Slack
PreviousNext
Powered by

kotlinlang

A modern programming language that makes developers happier.

Powered by