Hi folks, I’ve got a question about the KMP Gradle plugin. On my team we’re running into memory issu...

# gradle

[kevin.cianfarini]

Hi folks, I’ve got a question about the KMP Gradle plugin. On my team we’re running into memory issues, particularly when doing IDE sync. Our Gradle daemon routinely runs out of memory with a max heap size of 8g. I’ve done some JFR profiling of the daemon while syncing to see where hotspots might be and it looks like some calls to resolve dependencies for Native libraries in the KMP plugin are allocating a lot of memory.

Should I be expecting this many allocations for native dependency resolution? Our project is ~660 modules, most of which are KMP modules. We don’t see similar allocation totals for the JVM dependency resolvers. It looks like these allocations bottom out in reading properties files from klibs.

What’s a reasonable amount of memory consumption to expect from a 660 module project for KMP?

[kevin.cianfarini]

I’d be happy to post the JFR here if it helps anyone from Jetbrains optimize memory allocations. These memory issues have started to become a really big impediment to our work. Most people can’t sync the IDE at all without getting OOM, even when bumping xmx to 12g.

[kevin.cianfarini]

I've done some more digging, and a lot of the memory consumption for several different call frames seems to come from reading from the properties files here. Are there any optimizations we can make here, like perhaps doing some memorization of these properties files by project?

[tapchicoma]

Could you open a new Kotlin issue with all your findings?

[kevin.cianfarini]

I'm about to head on vacation unfortunately, but I'll try to get someone else to do that. Thanks!

[kevin.cianfarini]

Is it okay to open an issue that doesn't have any suggested remediation? We're still digging in, and based on our findings I'm not seeing any clear places memorization would help.

[tapchicoma]

Is it okay to open an issue that doesn't have any suggested remediation?

Yes, we will check from our side what could be done here

[Richard Leggett]

So we kept going down this path and found that for each module IdeMultiplatformImportFactory was adding ~17 resolvers, running for each of the 10 source sets that are applied by the default target hierarchy (commonMain/Test, appleMain/Test, iOS***Main/Test , nativeMain/Test )

Heap use dropped from 8-10gig to ~5.5gig by using kotlin.mpp.applyDefaultHierarchyTemplate=false with the experimental applyHierarchyTemplate {} applying only iOS and jvm() targets to most modules, and androidTarget() to UI modules.

This came with a bunch of config + build time benefits.

[Richard Leggett]

(I should note that almost all source is in commonMain, a couple of projects/modules use jvmMain and a handful use androidMain and iosMain - no other source sets are used).

[tapchicoma]

fyi @Anton Lakotka [JB]

[kevin.cianfarini]

Thanks for following through on this Rich 😄

[tapchicoma]

still could you create a Kotlin issue?:)

[kevin.cianfarini]

Rich, it might also be helpful to run a JFR profile with only the JVM target enabled with the default hierarchy as a baseline before you file that issue

[Richard Leggett]

@tapchicoma Indeed, I'm a little snowed under after last week but I've made a note to get a reproducer up.

[Richard Leggett]

Rich, it might also be helpful to run a JFR profile with only the JVM target enabled with the default hierarchy as a baseline before you file that issue

Yeah, IIRC I did this and you got saw the expected drop, heap use was peaking at like 3.5/4g.

[Anton Lakotka [JB]]

Can you also roughly draw size of your projects?
i.e. how many subproject do you have, how many source sets per subproject and how many dependencies per source set.

I missed this
> Our project is ~660 modules, most of which are KMP modules.
That's enough.

We're going to add this into our performance tests to measure memory consumption as well.

From what I see in screenshots there is lack of cache for Commonized Native Distribution.
i.e. for each sourceSet we independently re-run the same IdeCommonizedNativePlatformDependencyResolver over and over.
In fact it should be enough to read those dependencies once per Import session.

[Richard Leggett]

Would you still like an issue raised from us @Anton Lakotka [JB] or is this OK? This is the change we made in terms of showing the impact (note we only put code in iosMain() for any native targets)..

[kevin.cianfarini]

Rich, I’d like to still file an issue that includes a JFR profile of our project and helps illustrate the problem.

[Richard Leggett]

We've gone ahead and raised one here. Thanks again.

[kevin.cianfarini]

Following up on this issue. We’ve worked around this by specifying a custom KMP target hierarchy, but that’s only bought us some time. We’re wondering when we should expect this issue to be prioritized? Some of our developers are running into Gradle daemon OOM during IDE sync again.

[tapchicoma]

we will try to re-prioritize it

[Richard Leggett]

Thank you so much Yahor 🙏

[kevin.cianfarini]

Thank you so much 🎉

[kevin.cianfarini]

Critical severity! Thank you so much 🤯