I’ve started to notice some very long KSP task times in our app, and I’m wondering if anyone else has similar problems or insight into the root cause. One task in particular is taking up to 9 minutes (!!) on CI. However, sometimes it is as short as 1-2 minutes. There seems to be a huge variability in time. This is the ksp task for our root application module, which depends on all of the other modules in our project (~1800). Given that, it does have a lot to process, but it seems like it should be more consistent. Our builds are ephemeral, so daemon’s aren’t reused, but build cache is. I’m wondering if perhaps these long ksp times happen if there is extreme memory pressure. We give the kotlin daemon 50gb, but it is quite hard to get visibility into the memory usage of kotlin/ksp. Does anyone have suggestions for where I might want to look to optimize this? I’m hoping k2 helps of course, but it seems like there is something else going on here as well

Does increasing / decreasing heap size make it harder / easier to reproduce?

I could try decreasing the heap we give it and see what happens. we’re already maxing out our machines so i can’t give it much more unfortunately

Have you tried this? https://blog.jetbrains.com/kotlin/2022/06/introducing-kotlin-build-reports/

I’ve used it before to profile incremental behavior, but my understanding is that it doesn’t show any information about memory usage or ksp. Am I wrong about that? From the sample output on that page it doesn’t look to me like that information would be very helpful for debugging ksp performance

Not currently. It'll be quicker to add in the processors if you have access to most of them.

I’ve found that the culprit seems to be Dagger. We started using Dagger via KSP from its recent release, but performance could potentially be an issue for us now. We do give it lots of heap, and I haven’t verified yet to what extent the issue is worse when there is memory pressure. As a workaround, we leverage dagger reflect in most builds to avoid the codegen. If I have time I’ll profile the dagger ksp step and try to report to dagger owners

If you were running dagger via kapt, there could be added heap pressure after switching to ksp. IIRC kapt runs in process isolation in gradle worker. Moving to ksp means moving to KotlinCompileDaemon and it has to share the heap with other compilation tasks, because they can run in parallel. This may also explain why it is not steadily reproducible.

thank you!

What JVM settings are you using besides heap? 50 GB sounds like too much (unless Android lint is in the mix)

thanks for the pointers, I appreciate it. These are the settings we use for a 32 core, 120GB machine on CI

-Dorg.gradle.jvmargs="-Xmx30720m -Xss8m -Dkotlin.daemon.jvmargs=Xmx49830m -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=67 -XX:G1MaxNewSizePercent=67 -XX:MaxMetaspaceSize=8192m -XX:ReservedCodeCacheSize=128m"

for lint we give more heap to the gradle daemon than kotlin daemon. I’m not familiar at all with the memory waste you mentioned due to non compressed memory, but if we do less memory we get OOMs and daemons randomly dying; 12gb wouldn’t work at all for us. We have a lot of code unfortunately. We’re on jdk 17, but hopefully will be on 21 before long. I’ll read up on the settings you suggested 🙏

I’d recommend watching https://www.infoq.com/presentations/JVM-Performance-Tuning-twitter/ Specifically at 13:09 he talks about compressed object pointers and the memory waste I’m referring to. Once you go past 32GB RAM you can’t use compressed pointers anymore. The lack of compression means you need more memory to handle your memory allocations and thus the big jump to 48GB

we have

XX:MaxMetaspaceSize=8192m

doing that already, right?

JDK 17 -> 19 was a pretty great perf improvement for all algos (parallel + g1gc). JDK 20 landed some good things for G1GC https://tschatzl.github.io/2023/03/14/jdk20-g1-parallel-gc-changes.html

thanks a ton for the pointers, I’ll try those out

I appreciate it, taking a look 👀

So, I thought about this a lot because all I really had for years was CircleCI memory usage which really tells you nothing, but I was a solo dev so I never had time to look at better tooling. Stuff I’ve been looking at recently is and https://github.com/gradle/gradle-profiler. Haven’t been able to get the former working and honestly don’t see the reason to anymore now that I have config cache reliably working on CI. The latter has been interesting because it has a chrome-tracer plugin that Jake blogged about years ago. Got that working in my Android CI test repo this weekend and I imagine exploring the memory usage overtime matched up with the Gradle task tracing could be helpful.

cool, yeah, that’s what we had in mind to do eventually