would anyone be able to give me some advice for trying to debug why the Kotlin daemon uses so much e...

# gradle

[james]

would anyone be able to give me some advice for trying to debug why the Kotlin daemon uses so much extra RAM and keeps dying after bumping from Kotlin 1.6.21 to 1.7.10? no problems on my local machine, but on our CI machines we cannot build with Kotlin 1.7.10.. build dies every time with: Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed)

I’ve attached a screenshot of the resource usage up until the build dies.. and for some further context, when I look back at older jobs using Kotlin 1.6.21 I see their memory usage on the same graph peaks at around 65%. now it hits 100% and then dies. could this be a memory leak in the build phase? I’m not able to pull any useful logs either, even with enabling most kotlin.build.report.* options 😕

[tapchicoma]

Try to collect Kotlin daemon logs here: <system_tmp>/kotlin-daemon.*.log and probably also Gradle daemon logs here: <gradle_user_home>/daemon/<build_gradle_version>/*.log

[tapchicoma]

also you could add to org.gradle.jvmargs and kotlin.daemon.jvmargs additional parameter -XX:+HeapDumpOnOutOfMemoryError

[james]

thanks Yahor 🙏

I have HeapDumpOnOutOfMemoryError in both gradle and kotlin daemon args lists already, but I will try and see if I can pull the logs you mentioned

[tapchicoma]

do you see any heap dumps after build failure?

[james]

☝️ this is the dead daemon I log I just found in /tmp/kotlin-daemon.log

[james]

I’ll check ~/.gradle now

[tapchicoma]

Looks like Gradle daemon was killed or died with OOM

[james]

that’s what I wondered, but shouldn’t I get a heap dump if it’s OOM?

[tapchicoma]

OS could kill it on memory pressure. In this case you will not see heap dump

[james]

I’ve got the following in my gradle.properties :

org.gradle.jvmargs=-XX:+HeapDumpOnOutOfMemoryError -XX:+UseParallelGC -Dfile.encoding=UTF-8

kotlin.daemon.jvm.options=-Xmx4096m -XX:+HeapDumpOnOutOfMemoryError -XX:+UseParallelGC

[tapchicoma]

try to limit also Gradle daemon max heap size

[james]

oh and not shown in gradle props is the overall CI JAVA_OPTs heap limit set to 4g

[james]

any tips on gradle daemon max heap size? if my kotlin daemon limit is 4gb, my understanding is the gradle daemon will usually be okay with a smaller limit.. I know everyone’s configuration and circumstances will differ, but is that generally accurate advice to follow?

[tapchicoma]

on your image I see docker container has overall 6Gb of memory. Gradle + Kotlin daemon with current settings may exceed it.

[tapchicoma]

btw what is CPU count on your CI?

[james]

this particular workflow uses an executor with 3 CPU / 6 GB RAM

I could bump it up to the next size, but that means extra $$, and it works fine with Kotlin 1.6.21, so I’m just trying to understand what might have changed

[tapchicoma]

also you could use VisualVM (or similar tool) to observe heap consumption in typical scenario. Also to make a heap dump

[tapchicoma]

this particular workflow uses an executor with 3 CPU / 6 GB RAM

Just a guess - try to set org.gradle.workers.max=3 and run your CI build

[james]

that works! ..but.. the build takes ~10 min instead of the 6 min it previously took

I was also doing some more digging through logs here and I noticed this:
2022-09-27 11:31:34.165 [compiler] INFO: getDaemonJVMOptions: DaemonJVMOptions(maxMemory=2048m, maxMetaspaceSize=, reservedCodeCacheSize=, jvmParams=[-Djava.awt.headless=true, -D$java.rmi.server.hostname=127.0.0.1, -Dkotlin.environment.keepalive, -ea, Dkotlin.environment.keepalive, ea])

that’s from the Kotlin Daemon log.. but in those options I don’t see any of my options that are set in my gradle.properties file?

[tapchicoma]

Kotlin daemon heap size could be increased dynamically

[tapchicoma]

that works! ..but.. the build takes ~10 min instead of the 6 min it previously took

Each additional worker increases required heap size. Try to play with this value and set it only for CI builds depending on the docker image parameters.