I'm at the point where my team has ~150 Kotlin gra...

# gradle

[spierce7]

I'm at the point where my team has ~150 Kotlin gradle modules in my companies mono-repo application. It's at the point where it's annoyingly long to sync new dependencies or resolve build.gradle.kts changes in the app.

I made a gradle plugin using Kotlin over the weekend, and I was surprised to see things like extensions created via reflection, and things like the project still being `@Inject`ed into the constructor. Isn't using reflection to create abstract classes and reflection based dependency injection part of the problem why resolving the project configuration takes so long?

Is there any hope for performance in the next few years, as we will surely surpass 200 multiplatform Kotlin modules (which are perhaps heavier than a normal module due to their many configurations?), or do we need to start considering breaking up things into separate repositories?

[Gavin Ray]

Gradle 8.0 has some significant performance improvements -- one change (declarative plugins {} blocks) can improve build times by 20% if you're able to use it.

Have you tried using 8.0-milestone-5 as your Gradle version, and enabling the below:

# org.gradle.caching=(true,false)
# When set to true, Gradle will reuse task outputs from any previous build, when possible, resulting in much faster builds
org.gradle.caching=true

# org.gradle.configureondemand=(true,false)
# Enables incubating configuration on demand, where Gradle will attempt to configure only necessary projects. Default is false.
org.gradle.configureondemand=true

# org.gradle.parallel=(true,false)
# When configured, Gradle will fork up to org.gradle.workers.max JVMs to execute projects in parallel.
# To learn more about parallel task execution, see the section on Gradle build performance. Default is false.
org.gradle.parallel=true

# The org.gradle.jvmargs Gradle property controls the VM running the build.
# It defaults to -Xmx512m "-XX:MaxMetaspaceSize=256m"
#
# (The important one here is -XX:+TieredCompilation -XX:TieredStopAtLevel=1)
org.gradle.jvmargs=-Xmx2g -XX:MaxMetaspaceSize=512m -XX:+HeapDumpOnOutOfMemoryError -Xverify:none \
                   -XX:+UseParallelGC -XX:+TieredCompilation -XX:TieredStopAtLevel=1 -Dfile.encoding=UTF-8

# <https://kotlinlang.org/docs/whatsnew17.html#a-new-approach-to-incremental-compilation>
kotlin.incremental.useClasspathSnapshot=true

[Gavin Ray]

See the section here titled `Imporved Script compilation performance`:
• https://docs.gradle.org/8.0-milestone-5/release-notes.html#new-features-and-usability-improvements

[spierce7]

org.gradle.caching=true caches aren't enabled by default?

[Gavin Ray]

The above is the config we use at my org and it works reasonably well for our multi-module build that uses buildSrc and is based on convention-plugins. We don't have nearly that many that modules though. You may need a lot more memory in the jvmargs section but the rest of that should all help considerably

[spierce7]

I'm pretty sure org.gradle.parallel=true has been the default for a few years as well

[spierce7]

either yes or I'm gravely mistaken

[Gavin Ray]

Caching is not enabled by default, it can cause some difficult-to-debug errors:
• https://docs.gradle.org/current/userguide/build_cache.html#sec:build_cache_enable
By default, the build cache is not enabled. You can enable the build cache in a couple of ways:

[spierce7]

oh - it appears we have this on anyways

org.gradle.jvmargs=-Xmx4g -Xms1g -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
org.gradle.parallel=true
org.gradle.caching=true

[Gavin Ray]

I would add -XX:+UseParallelGC -XX:+TieredCompilation -XX:TieredStopAtLevel=1 to your jvmargs -- that'll only run a single JIT pass during development, so it'll be significantly faster. The Parallel GC is optional but usually a decent default

You may also look into enabling org.gradle.configureondemand=true so that tasks only fire when necessary

[Gavin Ray]

Hopefully those two should net some significant wins for you

[spierce7]

that'll only run a single JIT pass

Because of the daemon though, isn't it ok to let it just do it's thing?

[spierce7]

The sync performance issues happen on warm daemons

[Gavin Ray]

Ah you have a good point, I'm not sure if that will actually impact sync code since there's no compilation happening

[Vampire]

Configurations times will greatly improve once the configuration cache is stable (which should happen in Gradle 8 hopefully) and you have your build in a shape that it can use it. Then configuration time is down to nearly zero if you rerun a task you already run during the cache period

[Gavin Ray]

Yeah I would highly recommend at least changing your wrapper to the below and testing what the difference is like:

distributionUrl=https:\//services.gradle.org/distributions/gradle-8.0-milestone-5-bin.zip

[Gavin Ray]

Worst case, if something breaks, just roll it back 🤷

[Gavin Ray]

(We've been using Milestone 2 at my workplace and it's been solid/no issues thus far)

[eygraber]

My app has 300-400 modules. Configuration cache is the only way I stay sane.

[spierce7]

I haven't noticed an improvement from the configuration cache yet

[eygraber]

Then there might be some other issue. Our configuration time went from 30-60 seconds to sub second (after the initial run of the task).

[Vampire]

Did you enable it?

[mbonnin]

Configuration cache is helping a ton for us too (when it works 🙃 )

[mbonnin]

Back to the initial question, I'm sharing the sentiment that there does seem to be a lot of reflection in the configuration path

[mbonnin]

I'd be super curious about an analysis about what takes multiple seconds/minutes during configuration. From afar, configuration is "only" creating a task graph, not even reading that many files or compiling anything (besides the cached build.gradle scripts). Sounds like it should be fast?

[Vampire]

Depends on the project, we have significant configuration times too.
But the project is organically grown with pre-1.0 Gradle and not too nicely done. 😄