https://kotlinlang.org logo
#gradle
Title
# gradle
s

Simon Marquis

09/29/2023, 12:18 PM
👋 I'm trying to understand a strange behaviour of the Kotlin Compiler daemon. In our multi-module project, we depend on an included build that contains our convention plugins:
Copy code
// settings.gradle.kts

pluginManagement {
    includeBuild("build-logic")
}
When running a classic build, a new Kotlin Compiler daemon is spawn to compile the
build-logic
Kotlin classes, as expected. But when Gradle moves to the actual project compilation, a new Kotlin Compiler daemon is spawn again, and the previous one is not reused, nor killed. This is unexpected to me. By looking at the JVM arguments and system properties of both daemons (with VisualVM), they are almost identical. The only difference is on
java.class.path/sun.java.command
where on one hand it references Kotlin jars in version
1.9.0
versus
1.9.10
on the other. Is this the reason why the daemon is not reused? And if yes, how can this be changed? By looking at these docs:
Avoid specifying a version for the
kotlin-dsl
plugin
And the corresponding changelogs seems empty. So I'm not sure what to do about this, and would love to get some help 🙂
v

Vampire

09/29/2023, 12:42 PM
If that is the case, then it is probably because you productively use Kotlin 1.9.10, while the Gradle built-in Kotlin version for the build logic is 1.9.0. You could downgrade your project Kotlin version to 1.9.0, then it might be reused. Like it is, you use two different Kotlin version, so you have two different compiler daemons.
s

Simon Marquis

09/29/2023, 1:39 PM
Indeed, we rely on 1.9.10 in our project. Forcing the
kotlin-dsl
to 4.1.2 made the daemon reused and does not seem to introduce any regression 👍
v

Vampire

09/29/2023, 1:40 PM
You should never do that though
s

Simon Marquis

09/29/2023, 1:40 PM
😬
could this break the build-cache or the configuration-cache?
v

Vampire

09/29/2023, 1:44 PM
This can break many things. Gradlew controls the Kotlin version for build logic. It also forces the shipped Kotlin libs. So you end up with a mix of 1.9.0 and 1.9.10 for the build logic and can get quite some subtle or not so subtle problems.
s

Simon Marquis

09/29/2023, 1:46 PM
Understood. If I downgrade the build logic (included build and main projects) to use Kotlin 1.9.0 would I still be able to use Kotlin 1.9.10 for the app code, or do they all need to be aligned?
v

Vampire

09/29/2023, 1:48 PM
You did have 1.9.0 in the build logic before. And yes, you can just fine use 1.9.10 for the production code. You just do not get compile daemon reuse.
Build logic and production code are two separate things, they don't need to be aligned.
s

Simon Marquis

09/29/2023, 1:56 PM
I think I am missing one piece of the puzzle here 😅. To me, there are at least 3 places where I can configure Kotlin versions: • the included build (aka my
build-logic
) which is requested by
kotlin-dsl
plugin (1.9.0 in this case) • the main build (aka, all by
gradle.kts
files of all my other modules) which comes from the root
build.gradle.kts
(1.9.10 in this case)
Copy code
plugins {
  alias(libs.plugins.kotlin.jvm) apply false
}
• the app source code (aka
kotlin-stdlib*
) which comes from the applied plugin Or are these last two items always identical?
v

Vampire

09/29/2023, 1:58 PM
Sure, what you apply there is the plugin that cares about building your production code
s

Simon Marquis

09/29/2023, 1:59 PM
Ok, then I'm still not sure how I can downgrade the build version but at the same time keeping a more recent version in source code 🤔
v

Vampire

09/29/2023, 2:01 PM
You don't need to downgrade anything.
build-logic
uses 1.9.0 as defined through
kotlin-dsl
. Executing the build scripts of the main build uses 1.9.0 as defined through Gradle. The 1.9.10 plugin you apply makes your production classes compiled with 1.9.10.
s

Simon Marquis

09/29/2023, 2:03 PM
Executing the build scripts of the main build uses 1.9.0 as defined through Gradle.
I think this is the part that I don't get 😅
v

Vampire

09/29/2023, 2:08 PM
The Kotlin plugin version you apply has nothing to do with the runtime environment the buidscript is running in. The buildscripts are running with the bundled Kotlin version. The Kotlin plugin runs on that and sets up tasks that compile and run your production code with 1.9.10.
You can compare this to running Gradle with a Java 8 JDK, but you JVM toolchains to compile your production code with Java 21.
While Gradle and the build logic is running with Java 8, it sets up tasks that compiles, tests, runs, ... your actual production files using a Java 21 JDK
s

Simon Marquis

09/29/2023, 2:11 PM
Ok, but where does the embedded Kotlin version running the main build scripts comes from? Gradle? Because according to this table, Gradle 8.3 uses Kotlin 1.9.0. But we I run my build, the Kotlin Compile Daemon mentions Kotlin 1.9.10 jars.
v

Vampire

09/29/2023, 2:13 PM
You said you have two Kotlin compile daemons, one for the build logic using 1.9.0, one for the production code using 1.9.10, which is exactly what I described. Yes, it is comding from Gradle, the Kotlin runtime for the build logic is built-in in the Gradle version you are using.
s

Simon Marquis

09/29/2023, 2:21 PM
I think I get it, but my main goal was to avoid having 2 daemons in the first place (for memory constraints reasons).
h

hfhbd

09/29/2023, 2:41 PM
As a workaround you can use Gradle 8.4 rc 1/2, which uses Kotlin 1.9.10 🤷🏻‍♂️
v

Vampire

09/29/2023, 2:49 PM
my main goal was to avoid having 2 daemons
Well, for that you have to align the versions and thus downgrade your production Kotlin version to 1.9.0, or use Gradle 8.4, yes. But imho you really shouldn't couple these versions.
s

Simon Marquis

09/29/2023, 3:12 PM
Then there might be alternative workarounds that I am not aware of, because we keep raising the amount of ram needed, but it's hard to keep up with that pace (we are currently maxing out at 32g...). One could be to extract this
build-logic
into a dedicated project, and publish it somewhere to not have to compile it from the main project? Another one that I'm thinking about is, would it be possible to override some kind of Kotlin daemon timeout (time after which it is killed if unused)? Because at the moment, it is kept alive for at least the whole duration of the build, and hangs onto ~1 to 2g of RAM.
--no-daemon
does not seem to affect it, and
gradle.properties
inside the
build-logic
are ignored 🤷
v

Vampire

09/29/2023, 3:37 PM
Yes for 1. No idea for 2.
--no-daemon
is just about the Gradle build daemon, could well be that the Kotlin compile deamon survives that.
s

Simon Marquis

09/29/2023, 3:50 PM
There seems to be a
--daemon-autoshutdownIdleSeconds=7200
given as argument to the Kotlin Compile Daemon. But this is probably not configurable (at the moment) https://youtrack.jetbrains.com/issue/KT-50510/Gradle-Add-the-possibility-to-change-daemon-autoshutdownIdleSeconds Anyway, thanks a lot for all these informations 🙏
👌 1
a

Alexander.Likhachev

10/01/2023, 11:02 PM
You could choose the in-process compiler execution strategy for the build-logic module, then it won’t require the daemon at all. That strategy doesn’t yet support incremental compilation, however if you don’t change the module too frequently, you’ll be well. https://kotlinlang.org/docs/gradle-compilation-and-caches.html#defining-kotlin-compiler-execution-strategy
v

Vampire

10/01/2023, 11:03 PM
But iirc, this is discouraged 🙂
a

Alexander.Likhachev

10/01/2023, 11:05 PM
Right, the current implementation of the in-process strategy is suboptimal, however we have plans to improve it
👌 1
s

Simon Marquis

10/02/2023, 2:04 PM
What do you mean by suboptimal? This is meant for CI only, so I suppose incremental compilation might no be that useful. But if it results in more flakiness or even more RAM consumption at the benefit of only not spawning an extra VM, that might not be worth.
a

Alexander.Likhachev

10/02/2023, 6:23 PM
Currently it’s creating a new class loader for each compilation task, that may cause higher memory consumption and much slower compilation because JVM shall load classes and perform JIT compilation for the compiler classes almost from scratch. So I won’t recommend using it for the main build as I consider it consists of at least a few modules. However, it’s ok to use it for
buildSrc
or the single
build-logic
module. It shouldn’t cause any problems in this case if you’re ok with not having incremental compilation for that module.
s

Simon Marquis

10/02/2023, 6:30 PM
Indeed we have ~1k modules on the main project 😅
Good to know anyway 👍
The embedded Kotlin has been upgraded to 1.9.10
https://docs.gradle.org/8.4/release-notes.html 👌
8 Views