Thread
#multiplatform
    Paul Woitaschek

    Paul Woitaschek

    11 months ago
    Are there any ongoing activities towards build optimizations? Our internal library has become quite big and is heavily modularized. However especially the sync time is extremely long (about 5-8 minutes) and that is even with multiple optimizations applied that completely disable all native targets through gradle properties. So this is measured on a pure jvm-android project.
    Mustafa Ozhan

    Mustafa Ozhan

    11 months ago
    are you using buildSrc or compositeBuilds ?
    Paul Woitaschek

    Paul Woitaschek

    11 months ago
    Composite builds
    Mustafa Ozhan

    Mustafa Ozhan

    11 months ago
    Ohh okay then to me composite builds makes difference but you already have it so no idea, following the thread now on 🙂
    Paul Woitaschek

    Paul Woitaschek

    11 months ago
    The only thing making it fast is disabling hmpp
    louiscad

    louiscad

    11 months ago
    How many modules do you have?
    Paul Woitaschek

    Paul Woitaschek

    11 months ago
    30ish
    louiscad

    louiscad

    11 months ago
    @Sebastian Sellmair [JB] Do you have any internal benchmarks/tests with that many multiplatform modules? If not, I think it'd be helpful to have one, to set our expectations and maybe provide a starting point for optimizations :blob-thinking-fast:
    Paul Woitaschek

    Paul Woitaschek

    11 months ago
    Iirc it's about 40k loc
    The major problem is the ide. Doing things on command line takes time but it's not these minutes of waiting. We've stopped creating new modules due to large sync times.
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    11 months ago
    5-8 minutes, when just creating a new module? 😮, with just jvm targets syncing? As described here, this is not acceptable at all to me. Are there ways to get access to that internal library? Is there a good OS project that also suffers in a similar manner?
    @louiscad, thanks for pinging me here!
    Paul Woitaschek

    Paul Woitaschek

    11 months ago
    @Sebastian Sellmair [JB] I can't share it unfortunately. I just measured it and it's no longer as bad as it used to be. I added a new empty module and it only takes 1:45 min to sync and then an additional 45 seconds to index
    When I flip hmpp to: kotlin.native.enableDependencyPropagation=true kotlin.mpp.enableGranularSourceSetsMetadata=false Its 45 seconds sync + 3 seconds indexing
    On the command line it doesn't seem to make any difference: ./gradlew jvmJar --rerun-tasks --no-build-cache 55 seconds without hmpp 56 with hmpp
    with just jvm targets syncing?
    jvm+android+commonJavaMain being the parent of both
    And it's 48 modules, not 30. And 32k loc, not 40k
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    11 months ago
    Okay, thanks for the info! Do you maybe roughly know what you’re waiting during syncing? What is intellij telling you it is doing? “Build gradle model”? Do you have any idea what it starts to re-index syncing?
    Paul Woitaschek

    Paul Woitaschek

    11 months ago
    Yep mostly building model
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    11 months ago
    Okay, so not running any Gradle tasks pre-sync, right? I think we can already work with that.
    Paul Woitaschek

    Paul Woitaschek

    11 months ago
    What do you mean by
    Okay, so not running any Gradle tasks pre-sync, right?
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    11 months ago
    During IDE sync, Gradle tasks might. be running, like custom artifact transformations, commonizer, or just preparation of the buildscript model.
    Paul Woitaschek

    Paul Woitaschek

    11 months ago
    Nope doesn't look like.
    Type-safe dependency accessors is an incubating feature.
    Type-safe project accessors is an incubating feature.
    > Task :plugins:extractPrecompiledScriptPluginPlugins UP-TO-DATE
    > Task :plugins:generateExternalPluginSpecBuilders UP-TO-DATE
    > Task :plugins:compilePluginsBlocks UP-TO-DATE
    > Task :plugins:generatePrecompiledScriptPluginAccessors UP-TO-DATE
    > Task :plugins:configurePrecompiledScriptDependenciesResolver
    > Task :plugins:generateScriptPluginAdapters UP-TO-DATE
    > Task :plugins:compileKotlin UP-TO-DATE
    > Task :plugins:compileJava NO-SOURCE
    > Task :plugins:pluginDescriptors UP-TO-DATE
    > Task :plugins:processResources UP-TO-DATE
    > Task :plugins:classes UP-TO-DATE
    > Task :plugins:inspectClassesForKotlinIC UP-TO-DATE
    > Task :plugins:jar UP-TO-DATE
    
    > Configure project :changesIndicator
    Unable to detect AGP versions for included builds. All projects in the build should use the same AGP version. Class name for the included build object: org.gradle.composite.internal.DefaultIncludedBuild$IncludedBuildImpl_Decorated.
    Warning: Kotlin language settings property 'experimentalAnnotationsInUse' is deprecated and will be removed in next major releases. Please, use 'optInAnnotationsInUse' instead.
    And it takes a long time indexing things
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    11 months ago
    Thanks a lot! I will hope being able to either tackle import performance myself in some reasonable time, or will nudge others 👀 I’ll keep you updated!
    louiscad

    louiscad

    11 months ago
    If you struggle to get a reproducer, keep in mind my open source project Splitties is one regarding slow IDE indexing.
    Paul Woitaschek

    Paul Woitaschek

    11 months ago
    Also with Kotlin poet you can easily set up a n-modules x-loc project yourself for Benchmarking
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    11 months ago
    Is poet a mpp library? 🤔
    louiscad

    louiscad

    11 months ago
    Not itself AFAIK, but it can write Kotlin code for any targets. It's used in SqlDelight for example.
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    11 months ago
    Hmm. I think I at least found one obvious way to reduce the number of libraries to index drastically, so that in theory adding a new module would not even require any indexing 👀
    Paul Woitaschek

    Paul Woitaschek

    11 months ago
    With poet you could create a script that creates a project with a defined number of modules each containing a defined number of classes, functions, whatnot. That would be my first approach for Benchmarking with multiple paramters
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    11 months ago
    Ahhh! Got what you meant 👌
    louiscad

    louiscad

    11 months ago
    A great motive to try the first EAP that would include this optimisation 👀
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    11 months ago
    I do not see a reason why adding a new module should take any longer than 10 seconds to sync.
    Paul Woitaschek

    Paul Woitaschek

    11 months ago
    One obvious way
    Can you describe what you have found? In curious
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    11 months ago
    louiscad

    louiscad

    11 months ago
    I think this would deserve a prominent before/after comparison in a blog post regarding the indexing speedup.
    Oooop, looks like I've been overhyped too fast 😅
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    11 months ago
    No, I would not propose to speedup indexing (though there seems to be potential improvements) But I would just try to avoid indexing here 👀
    louiscad

    louiscad

    11 months ago
    Well, technically, it's an infinite reduction in indexing time.
    Paul Woitaschek

    Paul Woitaschek

    11 months ago
    Four our project indexing is only a part of the story
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    11 months ago
    Where does it also hang?
    Any input will be appreciated!
    louiscad

    louiscad

    11 months ago
    Splitties usually also takes one minute for a Gradle sync, even without adding a module
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    11 months ago
    Long time between those Gradle tasks ands before indexing?
    Paul Woitaschek

    Paul Woitaschek

    11 months ago
    The building module thing
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    11 months ago
    Okay, got it!
    Paul Woitaschek

    Paul Woitaschek

    11 months ago
    How long does it take for splitties?
    louiscad

    louiscad

    11 months ago
    How long does it take for splitties?
    What exactly?
    Paul Woitaschek

    Paul Woitaschek

    11 months ago
    Sync+index
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    11 months ago
    Here is technical detail on how to avoid indexing after adding a new project: Right now, metadata (Kotlin headers) are shipped in this -all.jar artifact. This artifact is a composite artifact that includes the metadata compilation of multiple source sets of a Library (and a little more). Right now, a transformation “GranularMetadataTransformation” is running on those artifacts to extract the corresponding klibs from that artifact. Right now, those are scoped by Gradle project and will create new transformation artifacts per Gradle project. Those can be stored globally and then the same library can be used for analysis in the IDE. If it is already indexed, no reindexing will be required after adding a new module. Sorry for the very short explanation. If you folks are interested, I can give you a more in depth technical presentation :virtual-meeting: ? But for now: It’s weekend!!! 🎉 🥳
    Paul Woitaschek

    Paul Woitaschek

    11 months ago
    I have vacation, feel free to start you presentation now
    I'm in the garden with my kids, they would love it too!
    louiscad

    louiscad

    11 months ago
    Up to 10 minutes for almost 50 modules
    Paul Woitaschek

    Paul Woitaschek

    11 months ago
    Ouch
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    11 months ago
    Unfortunately, I have an appointment in a Bar right now 🙃 Feel free to join in Rosenheim, Nenas 😅
    I will very happily talk about indexing performance there
    Paul Woitaschek

    Paul Woitaschek

    11 months ago
    Really? We are going to Rosenheim on monday
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    11 months ago
    Not really??? Rosenheim, Germany, Bavaria?
    Wanna meet up then?
    Paul Woitaschek

    Paul Woitaschek

    11 months ago
    Yeah sure 😃 I'll drop you a pm
    Is there an issue that's tracking this?
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    6 months ago
    Not yet! I have implemented the optimisation in a PR for KPM, but not the current multiplatform implementation. How heavy are you suffering? Would you mind jumping into a call with me, to evaluate if i should spend resources to also implement this in the current multiplatform model for 1.7.0?
    Paul Woitaschek

    Paul Woitaschek

    6 months ago
    Suffering a lot 😉 I’d be available right now
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    6 months ago
    Oh, I see! Then would you mind opening intellij, preparing what you can show me and then drop me another message, then we can jump into a call 👍 👍
    Paul Woitaschek

    Paul Woitaschek

    6 months ago
    Yep, ide is open
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    6 months ago
    Sending link to join in a minute
    louiscad

    louiscad

    6 months ago
    @Sebastian Sellmair [JB] I see I participated in this old thread, should I also join?
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    6 months ago
    👍 Yes!
    Paul Woitaschek

    Paul Woitaschek

    6 months ago
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    6 months ago
    @Paul Woitaschek: Are you using cinterops in your project?
    Paul Woitaschek

    Paul Woitaschek

    6 months ago
    Nope
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    6 months ago
    Do you happen to have kotlin.git setup so that you could compile the compiler and kotlin gradle plugin=?
    Basically just clone kotlin.git and hit ./gradlew install
    Paul Woitaschek

    Paul Woitaschek

    6 months ago
    Yes I have git and java 😁
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    6 months ago
    😛arrot-upside-down: Then do it 😁 Checkout rra/sellmair/KT-48135-deduplicate-extracted-metadata-klibs Hit ./gradlew install And then in your project use kotlin version 1.7.255-SNAPSHOT and see if you can spot differences!
    I think one is still required to setup some environment variables (according to our readme). I scheduled a dev build also, so if you have challenges building kotlin here, the dev build should be available in an hour, when all goes according to plan.
    Paul Woitaschek

    Paul Woitaschek

    6 months ago
    Already on the playground, I'll report tomorrow and let you know
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    6 months ago
    I really hope, this will help us!!! Properly implementing this would require roughly one working day, which I think is absolutely justified if you can notice a difference 🤞
    Paul Woitaschek

    Paul Woitaschek

    6 months ago
    How do I profile this?
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    6 months ago
    This is how you would profile the idea part of syncing Just start the profiling, hit the sync button, stop and send me the file ☺️
    For profiling the Gradle process during sync, probably async profiler might do:https://github.com/jvm-profiling-tools/async-profiler
    Paul Woitaschek

    Paul Woitaschek

    6 months ago
    It fails with a tons of issues in the form of:
    e: /Users/ph1b/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-compiler-embeddable/1.7.255-SNAPSHOT/56df3d2fef65431fcaf99063e5ca2e44c9a786a1/kotlin-compiler-embeddable-1.7.255-SNAPSHOT.jar!/META-INF/wasm.ir.kotlin_module: Module was compiled with an incompatible version of Kotlin. The binary version of its metadata is 1.7.0, expected version is 1.5.1.
    
    > Task :core:compileKotlin FAILED
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    6 months ago
    You updated all kotlin dependencies accordingly to 1.7.255-SNAPSHOT? If so,. let me prepare a branch based from 1.6.20
    Paul Woitaschek

    Paul Woitaschek

    6 months ago
    Yep, it’s all defined in the version catalog. Maybe it’s related to ksp, that I cannot easily update
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    6 months ago
    I’ll send another branch later 👍
    Paul Woitaschek

    Paul Woitaschek

    6 months ago
    Building myself didn’t work on an m1
    > Could not read ’https://api.adoptopenjdk.net/v3/binary/latest/8/ga/mac/aarch64/jdk/hotspot/normal/adoptopenjdk' as it does not exist.
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    6 months ago
    Can you quickly check if 1.6.20-RC works for your project?
    Paul Woitaschek

    Paul Woitaschek

    6 months ago
    Yes it works
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    6 months ago
    I pushed a branch for 1.6.20:sellmair/1.6.20/yazio/kt-48135
    ./gradlew install -PdeployVersion=1.6.20-yazio
    And then change kotlin version to 1.6.20-yazio ☺️ 👍 Hehe, note: I did not test that branch. Also my machine is still compiling it, will drop another message if I push into that branch again. 😬
    Paul Woitaschek

    Paul Woitaschek

    6 months ago
    I installed jdk 8 now but now I think it needs even 6 🙈 > No compatible toolchains found for request filter: {languageVersion=6, vendor=any, implementation=vendor-specific} (auto-detect true, auto-download true)
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    6 months ago
    Need to fix merge issues anyways 😬 Will ping you once the branch is ready again. 👀
    Paul Woitaschek

    Paul Woitaschek

    6 months ago
    Can you push a CI branch later?
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    6 months ago
    Yes, I think so 👍
    However, CI build will take probably two hours 😕
    Paul Woitaschek

    Paul Woitaschek

    6 months ago
    Tomorrow is another day 🙂
    louiscad

    louiscad

    6 months ago
    Is it so long because the CI build also runs all tests?
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    6 months ago
    @Paul Woitaschek: Maybe this gradle property helps? kotlin.build.isObsoleteJdkOverrideEnabled=true
    dev build will be available roughly 17:30 The reason is that this dev builds also build a full native distribution, which takes quite some time
    But the property should work on an arm mac 👍
    CI is currently pretty busy, sorry folks! Dev builds will take until tomorrow 😕
    Paul Woitaschek

    Paul Woitaschek

    6 months ago
    So the sync feels kind of the same - the indexing feels way faster. I can’t profile it as these options don’t exist in AS: And I can’t use Intellij due to it being incompatible with agp 7.1
    Adding a new module and syncing still takes about 5 minutes
    louiscad

    louiscad

    6 months ago
    @Paul Woitaschek it's part of a plugin you can install, it's from JetBrains, let me find the name when I have my IDE in front of me.
    Sebastian Sellmair [JB]

    Sebastian Sellmair [JB]

    6 months ago
    Okay, I asked for a new channel for us #multiplatform-idea-performance@louiscad: Can we update splitties to 1.6.20 in one branch?
    louiscad

    louiscad

    6 months ago
    @Sebastian Sellmair [JB] Yup, I'll start updating to 1.6.10 first, and we can do it right after.
    Mayank

    Mayank

    6 months ago
    Hi 👋 I’m also facing the same issue and would love to help in any way possible. For me majority time is spent in “Build model…” step around 70-80% of total sync time. The project is using Kotlint 1.6.10 and has around 50 Kotlin multiplatform modules and 25 Android only modules. I should also mention the use of Cocoapods plugin in 2-3 modules. I can try the steps mentioned above, i.e. updating to 1.6.20 and using a dev build and report back with the updates. I’m hopeful that together we’ll be able to solve this issue and improve developer experience 😒imple_smile:
    Paul Woitaschek

    Paul Woitaschek

    6 months ago
    @Mayank Invited you
    Mayank

    Mayank

    6 months ago
    Thanks @Paul Woitaschek