https://kotlinlang.org logo
#multiplatform
Title
# multiplatform
p

Paul Woitaschek

10/13/2021, 2:49 PM
Are there any ongoing activities towards build optimizations? Our internal library has become quite big and is heavily modularized. However especially the sync time is extremely long (about 5-8 minutes) and that is even with multiple optimizations applied that completely disable all native targets through gradle properties. So this is measured on a pure jvm-android project.
5
m

Mustafa Ozhan

10/13/2021, 3:07 PM
are you using buildSrc or compositeBuilds ?
p

Paul Woitaschek

10/13/2021, 3:07 PM
Composite builds
m

Mustafa Ozhan

10/13/2021, 3:10 PM
Ohh okay then to me composite builds makes difference but you already have it so no idea, following the thread now on 🙂
p

Paul Woitaschek

10/13/2021, 3:19 PM
The only thing making it fast is disabling hmpp
👀 1
l

louiscad

10/14/2021, 2:17 PM
How many modules do you have?
p

Paul Woitaschek

10/14/2021, 2:19 PM
30ish
l

louiscad

10/14/2021, 2:20 PM
@Sebastian Sellmair [JB] Do you have any internal benchmarks/tests with that many multiplatform modules? If not, I think it'd be helpful to have one, to set our expectations and maybe provide a starting point for optimizations blob thinking fast
p

Paul Woitaschek

10/14/2021, 2:31 PM
Iirc it's about 40k loc
The major problem is the ide. Doing things on command line takes time but it's not these minutes of waiting. We've stopped creating new modules due to large sync times.
s

Sebastian Sellmair [JB]

10/14/2021, 3:51 PM
5-8 minutes, when just creating a new module? 😮, with just jvm targets syncing? As described here, this is not acceptable at all to me. Are there ways to get access to that internal library? Is there a good OS project that also suffers in a similar manner?
@louiscad, thanks for pinging me here!
🙂 1
p

Paul Woitaschek

10/14/2021, 7:03 PM
@Sebastian Sellmair [JB] I can't share it unfortunately. I just measured it and it's no longer as bad as it used to be. I added a new empty module and it only takes 1:45 min to sync and then an additional 45 seconds to index
When I flip hmpp to: kotlin.native.enableDependencyPropagation=true kotlin.mpp.enableGranularSourceSetsMetadata=false Its 45 seconds sync + 3 seconds indexing
On the command line it doesn't seem to make any difference: ./gradlew jvmJar --rerun-tasks --no-build-cache 55 seconds without hmpp 56 with hmpp
with just jvm targets syncing?
jvm+android+commonJavaMain being the parent of both
And it's 48 modules, not 30. And 32k loc, not 40k
s

Sebastian Sellmair [JB]

10/15/2021, 7:32 AM
Okay, thanks for the info! Do you maybe roughly know what you’re waiting during syncing? What is intellij telling you it is doing? “Build gradle model”? Do you have any idea what it starts to re-index syncing?
p

Paul Woitaschek

10/15/2021, 7:32 AM
Yep mostly building model
s

Sebastian Sellmair [JB]

10/15/2021, 7:33 AM
Okay, so not running any Gradle tasks pre-sync, right? I think we can already work with that.
p

Paul Woitaschek

10/15/2021, 7:34 AM
What do you mean by
Okay, so not running any Gradle tasks pre-sync, right?
s

Sebastian Sellmair [JB]

10/15/2021, 7:35 AM
During IDE sync, Gradle tasks might. be running, like custom artifact transformations, commonizer, or just preparation of the buildscript model.
p

Paul Woitaschek

10/15/2021, 7:35 AM
Nope doesn't look like.
Copy code
Type-safe dependency accessors is an incubating feature.
Type-safe project accessors is an incubating feature.
> Task :plugins:extractPrecompiledScriptPluginPlugins UP-TO-DATE
> Task :plugins:generateExternalPluginSpecBuilders UP-TO-DATE
> Task :plugins:compilePluginsBlocks UP-TO-DATE
> Task :plugins:generatePrecompiledScriptPluginAccessors UP-TO-DATE
> Task :plugins:configurePrecompiledScriptDependenciesResolver
> Task :plugins:generateScriptPluginAdapters UP-TO-DATE
> Task :plugins:compileKotlin UP-TO-DATE
> Task :plugins:compileJava NO-SOURCE
> Task :plugins:pluginDescriptors UP-TO-DATE
> Task :plugins:processResources UP-TO-DATE
> Task :plugins:classes UP-TO-DATE
> Task :plugins:inspectClassesForKotlinIC UP-TO-DATE
> Task :plugins:jar UP-TO-DATE

> Configure project :changesIndicator
Unable to detect AGP versions for included builds. All projects in the build should use the same AGP version. Class name for the included build object: org.gradle.composite.internal.DefaultIncludedBuild$IncludedBuildImpl_Decorated.
Warning: Kotlin language settings property 'experimentalAnnotationsInUse' is deprecated and will be removed in next major releases. Please, use 'optInAnnotationsInUse' instead.
And it takes a long time indexing things
s

Sebastian Sellmair [JB]

10/15/2021, 3:37 PM
Thanks a lot! I will hope being able to either tackle import performance myself in some reasonable time, or will nudge others 👀 I’ll keep you updated!
🙂 1
l

louiscad

10/15/2021, 3:39 PM
If you struggle to get a reproducer, keep in mind my open source project Splitties is one regarding slow IDE indexing.
p

Paul Woitaschek

10/15/2021, 3:41 PM
Also with Kotlin poet you can easily set up a n-modules x-loc project yourself for Benchmarking
s

Sebastian Sellmair [JB]

10/15/2021, 3:43 PM
Is poet a mpp library? 🤔
l

louiscad

10/15/2021, 3:44 PM
Not itself AFAIK, but it can write Kotlin code for any targets. It's used in SqlDelight for example.
s

Sebastian Sellmair [JB]

10/15/2021, 3:46 PM
Hmm. I think I at least found one obvious way to reduce the number of libraries to index drastically, so that in theory adding a new module would not even require any indexing 👀
🤩 3
p

Paul Woitaschek

10/15/2021, 3:48 PM
With poet you could create a script that creates a project with a defined number of modules each containing a defined number of classes, functions, whatnot. That would be my first approach for Benchmarking with multiple paramters
s

Sebastian Sellmair [JB]

10/15/2021, 3:48 PM
Ahhh! Got what you meant 👌
l

louiscad

10/15/2021, 3:49 PM
A great motive to try the first EAP that would include this optimisation 👀
s

Sebastian Sellmair [JB]

10/15/2021, 3:49 PM
I do not see a reason why adding a new module should take any longer than 10 seconds to sync.
p

Paul Woitaschek

10/15/2021, 3:49 PM
One obvious way
Can you describe what you have found? In curious
s

Sebastian Sellmair [JB]

10/15/2021, 3:50 PM
l

louiscad

10/15/2021, 3:50 PM
I think this would deserve a prominent before/after comparison in a blog post regarding the indexing speedup.
Oooop, looks like I've been overhyped too fast 😅
s

Sebastian Sellmair [JB]

10/15/2021, 3:50 PM
No, I would not propose to speedup indexing (though there seems to be potential improvements) But I would just try to avoid indexing here 👀
l

louiscad

10/15/2021, 3:51 PM
Well, technically, it's an infinite reduction in indexing time.
p

Paul Woitaschek

10/15/2021, 3:51 PM
Four our project indexing is only a part of the story
s

Sebastian Sellmair [JB]

10/15/2021, 3:52 PM
Where does it also hang?
Any input will be appreciated!
l

louiscad

10/15/2021, 3:52 PM
Splitties usually also takes one minute for a Gradle sync, even without adding a module
s

Sebastian Sellmair [JB]

10/15/2021, 3:52 PM
Long time between those Gradle tasks ands before indexing?
p

Paul Woitaschek

10/15/2021, 3:52 PM
The building module thing
s

Sebastian Sellmair [JB]

10/15/2021, 3:52 PM
Okay, got it!
p

Paul Woitaschek

10/15/2021, 3:52 PM
How long does it take for splitties?
l

louiscad

10/15/2021, 3:53 PM
How long does it take for splitties?
What exactly?
p

Paul Woitaschek

10/15/2021, 3:54 PM
Sync+index
s

Sebastian Sellmair [JB]

10/15/2021, 3:56 PM
Here is technical detail on how to avoid indexing after adding a new project: Right now, metadata (Kotlin headers) are shipped in this -all.jar artifact. This artifact is a composite artifact that includes the metadata compilation of multiple source sets of a Library (and a little more). Right now, a transformation “GranularMetadataTransformation” is running on those artifacts to extract the corresponding klibs from that artifact. Right now, those are scoped by Gradle project and will create new transformation artifacts per Gradle project. Those can be stored globally and then the same library can be used for analysis in the IDE. If it is already indexed, no reindexing will be required after adding a new module. Sorry for the very short explanation. If you folks are interested, I can give you a more in depth technical presentation virtual meeting ? But for now: It’s weekend!!! 🎉 🥳
👀 1
p

Paul Woitaschek

10/15/2021, 3:57 PM
I have vacation, feel free to start you presentation now
I'm in the garden with my kids, they would love it too!
l

louiscad

10/15/2021, 4:11 PM
Up to 10 minutes for almost 50 modules
p

Paul Woitaschek

10/15/2021, 4:15 PM
Ouch
s

Sebastian Sellmair [JB]

10/15/2021, 4:39 PM
Unfortunately, I have an appointment in a Bar right now 🙃 Feel free to join in Rosenheim, Nenas 😅
😄 2
I will very happily talk about indexing performance there
p

Paul Woitaschek

10/15/2021, 4:54 PM
Really? We are going to Rosenheim on monday
s

Sebastian Sellmair [JB]

10/15/2021, 8:02 PM
Not really??? Rosenheim, Germany, Bavaria?
Wanna meet up then?
p

Paul Woitaschek

10/16/2021, 9:32 AM
Yeah sure :) I'll drop you a pm
Is there an issue that's tracking this?
s

Sebastian Sellmair [JB]

03/08/2022, 10:01 AM
Not yet! I have implemented the optimisation in a PR for KPM, but not the current multiplatform implementation. How heavy are you suffering? Would you mind jumping into a call with me, to evaluate if i should spend resources to also implement this in the current multiplatform model for 1.7.0?
p

Paul Woitaschek

03/08/2022, 10:02 AM
Suffering a lot 😉 I’d be available right now
s

Sebastian Sellmair [JB]

03/08/2022, 10:02 AM
Oh, I see! Then would you mind opening intellij, preparing what you can show me and then drop me another message, then we can jump into a call 👍 👍
p

Paul Woitaschek

03/08/2022, 10:03 AM
Yep, ide is open
s

Sebastian Sellmair [JB]

03/08/2022, 10:04 AM
Sending link to join in a minute
l

louiscad

03/08/2022, 10:05 AM
@Sebastian Sellmair [JB] I see I participated in this old thread, should I also join?
s

Sebastian Sellmair [JB]

03/08/2022, 10:06 AM
👍 Yes!
p

Paul Woitaschek

03/08/2022, 11:07 AM
s

Sebastian Sellmair [JB]

03/08/2022, 3:32 PM
@Paul Woitaschek: Are you using cinterops in your project?
p

Paul Woitaschek

03/08/2022, 3:33 PM
Nope
s

Sebastian Sellmair [JB]

03/08/2022, 3:38 PM
Do you happen to have kotlin.git setup so that you could compile the compiler and kotlin gradle plugin=?
Basically just clone kotlin.git and hit ./gradlew install
p

Paul Woitaschek

03/08/2022, 3:49 PM
Yes I have git and java 😁
s

Sebastian Sellmair [JB]

03/08/2022, 4:01 PM
🦜 Then do it 😁 Checkout rra/sellmair/KT-48135-deduplicate-extracted-metadata-klibs Hit ./gradlew install And then in your project use kotlin version 1.7.255-SNAPSHOT and see if you can spot differences!
I think one is still required to setup some environment variables (according to our readme). I scheduled a dev build also, so if you have challenges building kotlin here, the dev build should be available in an hour, when all goes according to plan.
p

Paul Woitaschek

03/08/2022, 4:12 PM
Already on the playground, I'll report tomorrow and let you know
s

Sebastian Sellmair [JB]

03/08/2022, 4:13 PM
I really hope, this will help us!!! Properly implementing this would require roughly one working day, which I think is absolutely justified if you can notice a difference 🤞
p

Paul Woitaschek

03/08/2022, 4:22 PM
How do I profile this?
s

Sebastian Sellmair [JB]

03/08/2022, 6:51 PM
This is how you would profile the idea part of syncing Just start the profiling, hit the sync button, stop and send me the file ☺️
For profiling the Gradle process during sync, probably async profiler might do: https://github.com/jvm-profiling-tools/async-profiler
p

Paul Woitaschek

03/08/2022, 8:58 PM
It fails with a tons of issues in the form of:
Copy code
e: /Users/ph1b/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-compiler-embeddable/1.7.255-SNAPSHOT/56df3d2fef65431fcaf99063e5ca2e44c9a786a1/kotlin-compiler-embeddable-1.7.255-SNAPSHOT.jar!/META-INF/wasm.ir.kotlin_module: Module was compiled with an incompatible version of Kotlin. The binary version of its metadata is 1.7.0, expected version is 1.5.1.

> Task :core:compileKotlin FAILED
s

Sebastian Sellmair [JB]

03/09/2022, 8:19 AM
You updated all kotlin dependencies accordingly to 1.7.255-SNAPSHOT? If so,. let me prepare a branch based from 1.6.20
p

Paul Woitaschek

03/09/2022, 8:36 AM
Yep, it’s all defined in the version catalog. Maybe it’s related to ksp, that I cannot easily update
s

Sebastian Sellmair [JB]

03/09/2022, 8:43 AM
I’ll send another branch later 👍
p

Paul Woitaschek

03/09/2022, 8:43 AM
Building myself didn’t work on an m1
> Could not read ’https://api.adoptopenjdk.net/v3/binary/latest/8/ga/mac/aarch64/jdk/hotspot/normal/adoptopenjdk' as it does not exist.
s

Sebastian Sellmair [JB]

03/09/2022, 8:43 AM
Can you quickly check if 1.6.20-RC works for your project?
p

Paul Woitaschek

03/09/2022, 8:43 AM
Yes it works
s

Sebastian Sellmair [JB]

03/09/2022, 2:03 PM
I pushed a branch for 1.6.20: sellmair/1.6.20/yazio/kt-48135
./gradlew install -PdeployVersion=1.6.20-yazio
And then change kotlin version to 1.6.20-yazio ☺️ 👍 Hehe, note: I did not test that branch. Also my machine is still compiling it, will drop another message if I push into that branch again. 😬
p

Paul Woitaschek

03/09/2022, 2:09 PM
I installed jdk 8 now but now I think it needs even 6 🙈 > No compatible toolchains found for request filter: {languageVersion=6, vendor=any, implementation=vendor-specific} (auto-detect true, auto-download true)
s

Sebastian Sellmair [JB]

03/09/2022, 2:11 PM
Need to fix merge issues anyways 😬 Will ping you once the branch is ready again. 👀
p

Paul Woitaschek

03/09/2022, 2:12 PM
Can you push a CI branch later?
s

Sebastian Sellmair [JB]

03/09/2022, 2:12 PM
Yes, I think so 👍
However, CI build will take probably two hours 😕
p

Paul Woitaschek

03/09/2022, 2:16 PM
Tomorrow is another day 🙂
l

louiscad

03/09/2022, 2:17 PM
Is it so long because the CI build also runs all tests?
s

Sebastian Sellmair [JB]

03/09/2022, 2:20 PM
@Paul Woitaschek: Maybe this gradle property helps? kotlin.build.isObsoleteJdkOverrideEnabled=true
dev build will be available roughly 17:30 The reason is that this dev builds also build a full native distribution, which takes quite some time
But the property should work on an arm mac 👍
CI is currently pretty busy, sorry folks! Dev builds will take until tomorrow 😕
p

Paul Woitaschek

03/09/2022, 9:36 PM
So the sync feels kind of the same - the indexing feels way faster. I can’t profile it as these options don’t exist in AS: And I can’t use Intellij due to it being incompatible with agp 7.1
Adding a new module and syncing still takes about 5 minutes
l

louiscad

03/09/2022, 10:55 PM
@Paul Woitaschek it's part of a plugin you can install, it's from JetBrains, let me find the name when I have my IDE in front of me.
s

Sebastian Sellmair [JB]

03/10/2022, 8:31 AM
Okay, I asked for a new channel for us #multiplatform-idea-performance @louiscad: Can we update splitties to 1.6.20 in one branch?
l

louiscad

03/10/2022, 10:33 AM
@Sebastian Sellmair [JB] Yup, I'll start updating to 1.6.10 first, and we can do it right after.
m

Mayank

03/10/2022, 11:34 AM
Hi 👋 I’m also facing the same issue and would love to help in any way possible. For me majority time is spent in “Build model…” step around 70-80% of total sync time. The project is using Kotlint 1.6.10 and has around 50 Kotlin multiplatform modules and 25 Android only modules. I should also mention the use of Cocoapods plugin in 2-3 modules. I can try the steps mentioned above, i.e. updating to 1.6.20 and using a dev build and report back with the updates. I’m hopeful that together we’ll be able to solve this issue and improve developer experience simple smile
p

Paul Woitaschek

03/10/2022, 12:06 PM
@Mayank Invited you
m

Mayank

03/10/2022, 1:44 PM
Thanks @Paul Woitaschek
4 Views