https://kotlinlang.org
Join Slack
I’m wondering what I’m doing wrong when it comes to caching Gradle state between CI builds. Here’s a...
# github-workflows-kt
p
I’m wondering what I’m doing wrong when it comes to caching Gradle state between CI builds. Here’s an example: after making this change that doesn’t change any source code, still a full Gradle build occurred (see here). Since I use gradle-build-action, it states in its README that it does some caching. For the mentioned commit, I’d expect Gradle to complete within several seconds, inferring that no source code was changed (
UP-TO-DATE
). Any clues?
✅ 1
a
I've got no clue, but I have experienced something similar too
c
from the log it shows re-downloading Gradle itself; check that a current version of gradle-build-action is in use and it’s setup to cache Gradle distributions (this may be the default in recent versions)
And investigate this:
Gradle User Home already exists: will not restore from cache.
This may be related (touching something in ~/.gradle) prior to executing the build. https://github.com/gradle/gradle-build-action/issues/480
a
looking at the Build Scan it seems that Gradle Build Cache is disabled? https://scans.gradle.com/s/hp5aynbyslu3u/timeline?details=b2ro2byz5cuke
c
that’s perhaps related to the Gradle home caching being disabled (where pretty much everything is cached…):
• In the main step: 
Gradle User Home already exists: will not restore from cache.
• In the post-action: 
Cache will not be saved: not restored in main action step.
Even if not, no point in caching if it isn’t persisted between CI runs 😞
v
You should disable the "gradle" caching of the "setup-java" action
That interferes with the same of the Gradle build action
p
Thanks, I'll try it!
a
please report back here if it works
p
doesn't work 🤔 https://github.com/typesafegithub/github-workflows-kt/pull/885 although some of the warnings disappeared. How I infer it doesn't work: the runtime is similar
v
Did you run twice with the new settings?
The first run to prime the cache, the second run to use it
c
it looks like it was run multiple time; it’s now reporting that the cache is read-only and not saving entries.
Hmmm.
Cache is read-only: will not save state for use in subsequent builds.
It’s likely related to this behaviour of gradle-build-action - only updating caches from main branch.
v
Yeah, it also says so in the properties:
I'm not familar with that action, I prefer the 3rd party one, even though it misses support for the configuration cache iirc
c
Have used that action extensively but have never had a need to configure the cache-read-only / cache-write-only properties.
this will likely correct itself as this PR is merged into master; subsequent PRs (branches) will then start with the cache from master and delta from there.
Looking at my other uses of that action you’ll likely benefit (longer term) from adding 
gradle-home-cache-cleanup: true
which allows for cleaning up unused files in the Gradle home directory, such as when you upgrade a Gradle version, JDK toolchain version, etc. If using toolchains those JDKs aren’t automagically cached, configuration required:
Copy code
gradle-home-cache-includes: |
            jdks
            caches
            notifications
p
Oh, so it behaves differently for the main branch... Ok, let me merge the change and then see
Somehow I assumed that each branch gets its own cache, but it would be a bit unsafe for a default behavior
RTFM, Piotr 🤦
c
lol. we’ve all been there… If you do want each branch to behave that way it is configurable, but for PR branches not generally desirable. If there are long-lived (e.g. release) branches you could set those up to cache-write as well.
p
still doesn’t work as expected, I’m gonna revisit it one day, tracking in https://github.com/typesafegithub/github-workflows-kt/issues/890
a
you need to enable Build Cache, because without it Gradle can’t remember what’s up to date and what isn’t since the project is freshly checked out each time (right?)
c
Correct. Once build cache is enabled it will write it to Gradle home, which gradle-build-action will then further cache between CI runs.
even without that you’d still get benefit from gradle-build-action in caching dependencies (otherwise they’d be re-downloaded for each CI build)
p
💡 I somehow assumed it's the default behavior, and it isn't: https://docs.gradle.org/current/userguide/build_cache.html#sec:build_cache_enable
I guess it errs on the side of correctness VS performance
a
if you re-run the same task twice then Gradle will skip the work (so if you run 
./gradlew check
twice then the second task would instantly return ‘success’). But on a fresh checkout then Build Cache is needed to ‘remember’ any previous runs.
p
Trying it now
c
Interestingly 
gradle init
creates projects with the build cache enabled by default.
Copy code
# This file was generated by the Gradle 'init' task.
# <https://docs.gradle.org/current/userguide/build_environment.html#sec:gradle_configuration_properties>

org.gradle.parallel=true
org.gradle.caching=true
v
Iirc it is not so much correctness vs. performance, but performance vs. performance. I mean to remember that in certain builds it can happen that the caching degrades performance and for that reason it is not (yet?) enabled by default.
c
ah yea. recall something about fingerprinting inputs / outputs where there are Very Large numbers of them or somesuch.
p
BTW, have you guys ever tried Bazel? I haven’t, but the main attractor is that it’s artifact-based, not task-based, so it can cache more aggressively
c
The Gradle team did a comparison with Bazel a ways back. From my own playing with Bazel - it solves problems I didn’t have and didn’t offer enough benefit to migrate (which can be non-trivial).
If “more performance” is the goal there’s still the configuration-cache to enable. For that last run through the configuration phase is 50% of this job (albeit only 12s). `gradle-build-action`supports the configuration cache, saving/restoring it as with the build cache.
p
which build exactly is it? here the whole “Build” took 1m 57s, and I cannot request a scan, there’s some issue with Gradle
c
that was from the `build-for-UbunutuLatest`build scan published off the summary page.
v
The build scan works fine here
And 51.047s could have been saved by configuration cache in that build
Well, almost, ignoring the time to deserialize the cache entry
p
I’m eager to try out the configuration cache
v
Yes, do it, it is declared stable since 8.1 and is great. 🙂
Unless you use some plugins that are not yet compatible
p
is it again as simple as adding
Copy code
org.gradle.configuration-cache=true
?
c
yep, that’s the setting
v
Yes and no
If the build is already fully compatible to configuration cache, yes
p
(I’m on the go right now, will revisit later)
v
If not, then not
🙂
c
once enabled test as many real-world tasks/scenarios as possible locally, then repeat for CI (often CI runs tasks that are not / cannot be run locally).
p
Ok, it seems like more effort than I anticipated 😅 but thanks for the hints
v
Configuration cache is still a relatively young feature, so many plugins or patterns you might have copied are probably not yet compatible, especially configuration cache has some strict restrictions that are necessary.
On the pro-side I lied, you could have saved more time than I stated, because with CC enabled, all tasks can run in parallel if they don't depend on each other, even within one project.
a
I just add these to 
gradle.properties
Copy code
org.gradle.caching=true

org.gradle.unsafe.configuration-cache=true
org.gradle.unsafe.configuration-cache-problems=warn

org.gradle.parallel=true
and it always works fine (except with Kotlin Multiplatform projects, because that’s not Config Cache compatible yet) The signing plugin and Maven Central publishing aren’t compatible with config cache, but they can just be disabled as required by adding 
--no-configuration-cache
to the GitHub step, so the rest of the build still benefits
v
You should probably update your Gradle version. The properties should not have 
unsafe
anymore. And 
signing
and 
maven-publish
are compatible
Otherwise they probably wouldn't have declared the feature stable
c
Now that CC is stable the correct property is
Copy code
org.gradle.configuration-cache=true
Unclear if the unsafe ones still work in recent Gradle versions.
v
It works, but should be replaced with the new. I forgot whether it will emit a deprecation warning
a
Maven Central publishing isn’t compatible because Sonatype can’t handle parallel uploads 😬
v
But that has not much to do with CC, that is a problem since years and can easily be mitigated using the 
io.github.gradle-nexus.publish-plugin
plugin
a
it’s caused by CC because it’s the CC that enables the parallel uploads
v
If that is really the case even with mentioned plugin, then just make the publish tasks mutally exclusive by adding in a build-service with max-usage 1.
But as I said, this is then a problem since years, because if you only have one project to publish, CC shouldn't make a difference as it just allows multiple tasks to run in parallel. And with mutliple published projects it would already have been a problem with using 
--parallel
before.
Btw. @Chris Lee, regarding
Interestingly 
gradle init
creates projects with the build cache enabled by default.
It does not by default. Only if you tell it the not-default "yes" to the question about using experimental features. 🙂
c
ah yea, was wondering about that, lots of setup questions. thanks for clarifying.
The question isn’t around experimental features, its “new APIs and behaviour”:
Generate build using new APIs and behavior (some features may change in the next minor release)? (default: no) [yes, no] yes
a
Unclear if the unsafe ones still work in recent Gradle versions.
the 
.unsafe
variants still work and aren’t deprecated (yet) :) https://docs.gradle.org/8.2/userguide/upgrading_version_8.html#configuration_caching_options_renamed
690 Views
Open in Slack
PreviousNext
Powered by

kotlinlang

A modern programming language that makes developers happier.

Powered by