Are there any ongoing activities towards build optimizations? Our internal library has become quite big and is heavily modularized. However especially the sync time is extremely long (about 5-8 minutes) and that is even with multiple optimizations applied that completely disable all native targets through gradle properties. So this is measured on a pure jvm-android project.

are you using buildSrc or compositeBuilds ?

Composite builds

Ohh okay then to me composite builds makes difference but you already have it so no idea, following the thread now on 🙂

The only thing making it fast is disabling hmpp

How many modules do you have?

30ish

@Sebastian Sellmair [JB] Do you have any internal benchmarks/tests with that many multiplatform modules? If not, I think it'd be helpful to have one, to set our expectations and maybe provide a starting point for optimizations blob thinking fast

Iirc it's about 40k loc

5-8 minutes, when just creating a new module? 😮, with just jvm targets syncing? As described here, this is not acceptable at all to me. Are there ways to get access to that internal library? Is there a good OS project that also suffers in a similar manner?

@Sebastian Sellmair [JB] I can't share it unfortunately. I just measured it and it's no longer as bad as it used to be. I added a new empty module and it only takes 1:45 min to sync and then an additional 45 seconds to index

Okay, thanks for the info! Do you maybe roughly know what you’re waiting during syncing? What is intellij telling you it is doing? “Build gradle model”? Do you have any idea what it starts to re-index syncing?

Yep mostly building model

Okay, so not running any Gradle tasks pre-sync, right? I think we can already work with that.

What do you mean by

Okay, so not running any Gradle tasks pre-sync, right?

During IDE sync, Gradle tasks might. be running, like custom artifact transformations, commonizer, or just preparation of the buildscript model.

Nope doesn't look like.

Type-safe dependency accessors is an incubating feature.
Type-safe project accessors is an incubating feature.
> Task :plugins:extractPrecompiledScriptPluginPlugins UP-TO-DATE
> Task :plugins:generateExternalPluginSpecBuilders UP-TO-DATE
> Task :plugins:compilePluginsBlocks UP-TO-DATE
> Task :plugins:generatePrecompiledScriptPluginAccessors UP-TO-DATE
> Task :plugins:configurePrecompiledScriptDependenciesResolver
> Task :plugins:generateScriptPluginAdapters UP-TO-DATE
> Task :plugins:compileKotlin UP-TO-DATE
> Task :plugins:compileJava NO-SOURCE
> Task :plugins:pluginDescriptors UP-TO-DATE
> Task :plugins:processResources UP-TO-DATE
> Task :plugins:classes UP-TO-DATE
> Task :plugins:inspectClassesForKotlinIC UP-TO-DATE
> Task :plugins:jar UP-TO-DATE

> Configure project :changesIndicator
Unable to detect AGP versions for included builds. All projects in the build should use the same AGP version. Class name for the included build object: org.gradle.composite.internal.DefaultIncludedBuild$IncludedBuildImpl_Decorated.
Warning: Kotlin language settings property 'experimentalAnnotationsInUse' is deprecated and will be removed in next major releases. Please, use 'optInAnnotationsInUse' instead.

Thanks a lot! I will hope being able to either tackle import performance myself in some reasonable time, or will nudge others 👀 I’ll keep you updated!

If you struggle to get a reproducer, keep in mind my open source project Splitties is one regarding slow IDE indexing.

Also with Kotlin poet you can easily set up a n-modules x-loc project yourself for Benchmarking

Is poet a mpp library? 🤔

Not itself AFAIK, but it can write Kotlin code for any targets. It's used in SqlDelight for example.

Hmm. I think I at least found one obvious way to reduce the number of libraries to index drastically, so that in theory adding a new module would not even require any indexing 👀

With poet you could create a script that creates a project with a defined number of modules each containing a defined number of classes, functions, whatnot. That would be my first approach for Benchmarking with multiple paramters

Ahhh! Got what you meant 👌

A great motive to try the first EAP that would include this optimisation 👀

I do not see a reason why adding a new module should take any longer than 10 seconds to sync.

One obvious way

Can you describe what you have found? In curious

I think this would deserve a prominent before/after comparison in a blog post regarding the indexing speedup.

No, I would not propose to speedup indexing (though there seems to be potential improvements) But I would just try to avoid indexing here 👀

Well, technically, it's an infinite reduction in indexing time.

Four our project indexing is only a part of the story

Where does it also hang?

Splitties usually also takes one minute for a Gradle sync, even without adding a module

Long time between those Gradle tasks ands before indexing?

The building module thing

Okay, got it!

How long does it take for splitties?

How long does it take for splitties?

What exactly?

Sync+index

Here is technical detail on how to avoid indexing after adding a new project: Right now, metadata (Kotlin headers) are shipped in this -all.jar artifact. This artifact is a composite artifact that includes the metadata compilation of multiple source sets of a Library (and a little more). Right now, a transformation “GranularMetadataTransformation” is running on those artifacts to extract the corresponding klibs from that artifact. Right now, those are scoped by Gradle project and will create new transformation artifacts per Gradle project. Those can be stored globally and then the same library can be used for analysis in the IDE. If it is already indexed, no reindexing will be required after adding a new module. Sorry for the very short explanation. If you folks are interested, I can give you a more in depth technical presentation virtual meeting ? But for now: It’s weekend!!! 🎉 🥳

I have vacation, feel free to start you presentation now

Up to 10 minutes for almost 50 modules

Ouch

Unfortunately, I have an appointment in a Bar right now 🙃 Feel free to join in Rosenheim, Nenas 😅

Really? We are going to Rosenheim on monday

Not really??? Rosenheim, Germany, Bavaria?

Yeah sure :) I'll drop you a pm

Not yet! I have implemented the optimisation in a PR for KPM, but not the current multiplatform implementation. How heavy are you suffering? Would you mind jumping into a call with me, to evaluate if i should spend resources to also implement this in the current multiplatform model for 1.7.0?

Suffering a lot 😉 I’d be available right now

Oh, I see! Then would you mind opening intellij, preparing what you can show me and then drop me another message, then we can jump into a call 👍 👍

Yep, ide is open

Sending link to join in a minute

@Sebastian Sellmair [JB] I see I participated in this old thread, should I also join?

👍 Yes!

@Paul Woitaschek: Are you using cinterops in your project?

Nope

Do you happen to have kotlin.git setup so that you could compile the compiler and kotlin gradle plugin=?

Yes I have git and java 😁

🦜 Then do it 😁 Checkout rra/sellmair/KT-48135-deduplicate-extracted-metadata-klibs Hit ./gradlew install And then in your project use kotlin version 1.7.255-SNAPSHOT and see if you can spot differences!

Already on the playground, I'll report tomorrow and let you know

I really hope, this will help us!!! Properly implementing this would require roughly one working day, which I think is absolutely justified if you can notice a difference 🤞

How do I profile this?

This is how you would profile the idea part of syncing Just start the profiling, hit the sync button, stop and send me the file ☺️

It fails with a tons of issues in the form of:

e: /Users/ph1b/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-compiler-embeddable/1.7.255-SNAPSHOT/56df3d2fef65431fcaf99063e5ca2e44c9a786a1/kotlin-compiler-embeddable-1.7.255-SNAPSHOT.jar!/META-INF/wasm.ir.kotlin_module: Module was compiled with an incompatible version of Kotlin. The binary version of its metadata is 1.7.0, expected version is 1.5.1.

> Task :core:compileKotlin FAILED

You updated all kotlin dependencies accordingly to 1.7.255-SNAPSHOT? If so,. let me prepare a branch based from 1.6.20

Yep, it’s all defined in the version catalog. Maybe it’s related to ksp, that I cannot easily update

I’ll send another branch later 👍

Building myself didn’t work on an m1

> Could not read ’https://api.adoptopenjdk.net/v3/binary/latest/8/ga/mac/aarch64/jdk/hotspot/normal/adoptopenjdk' as it does not exist.

Can you quickly check if 1.6.20-RC works for your project?

Yes it works

I pushed a branch for 1.6.20: sellmair/1.6.20/yazio/kt-48135

./gradlew install -PdeployVersion=1.6.20-yazio

And then change kotlin version to 1.6.20-yazio ☺️ 👍 Hehe, note: I did not test that branch. Also my machine is still compiling it, will drop another message if I push into that branch again. 😬

I installed jdk 8 now but now I think it needs even 6 🙈 > No compatible toolchains found for request filter: {languageVersion=6, vendor=any, implementation=vendor-specific} (auto-detect true, auto-download true)

Need to fix merge issues anyways 😬 Will ping you once the branch is ready again. 👀

Can you push a CI branch later?

Yes, I think so 👍

Tomorrow is another day 🙂

Is it so long because the CI build also runs all tests?

@Paul Woitaschek: Maybe this gradle property helps? kotlin.build.isObsoleteJdkOverrideEnabled=true

So the sync feels kind of the same - the indexing feels way faster. I can’t profile it as these options don’t exist in AS: And I can’t use Intellij due to it being incompatible with agp 7.1

@Paul Woitaschek it's part of a plugin you can install, it's from JetBrains, let me find the name when I have my IDE in front of me.

Okay, I asked for a new channel for us #multiplatform-idea-performance @louiscad: Can we update splitties to 1.6.20 in one branch?

@Sebastian Sellmair [JB] Yup, I'll start updating to 1.6.10 first, and we can do it right after.

Hi 👋 I’m also facing the same issue and would love to help in any way possible. For me majority time is spent in “Build model…” step around 70-80% of total sync time. The project is using Kotlint 1.6.10 and has around 50 Kotlin multiplatform modules and 25 Android only modules. I should also mention the use of Cocoapods plugin in 2-3 modules. I can try the steps mentioned above, i.e. updating to 1.6.20 and using a dev build and report back with the updates. I’m hopeful that together we’ll be able to solve this issue and improve developer experience simple smile

@Mayank Invited you

Thanks @Paul Woitaschek