I’m saving in GItlab CI cache folder: `.gradle/caches/` which rapidly grows from 40k to 400k files d...

# gradle

[Nick Kleban]

I’m saving in GItlab CI cache folder: .gradle/caches/ which rapidly grows from 40k to 400k files during a day. And saving\restoring cache takes quite a lot of time. Is there any sensible way to optimise stored cache?

[ephemient]

https://docs.gradle.org/current/userguide/dependency_management.html#sub:ephemeral-ci-cache

[CLOVIS]

You should also look into enabling the Gradle Build Cache, which is the only efficient way to do caching between CI jobs.

If you are using shared runners, the runners have to download everything when they start. Whether they download from your shared cache or from the original source doesn't change much. If that's your case, using a download cache won't make the build faster.

If you use your own runners, caching the Gradle wrapper will make the build faster without it growing exponentially: https://gitlab.com/opensavvy/ci-templates/-/blob/main/gradle.gitlab-ci.yml

[ephemient]

if e.g. Maven Central is the same speed as your local network, then reusing dependency cache shouldn't make a difference, but it should be likely that you have a faster path to wherever you're saving/restoring cache than the external Internet, and that will save some time. but it shouldn't be a major contributing factor to your build speeds unless your Internet is really slow

[ephemient]

build cache can definitely help but I don't believe that copying it around is a supported way of reusing it.

[CLOVIS]

^ that's completely true, but keep in mind that each library that's in the cache that you don't end up using is wasted network IO. It takes surprisingly few unused libraries in the download cache to make extraction take more time than downloading everything from Central would've taken.

I've seen a project where removing caching of the `~/.gradle`folder altogether led to CI being twice as fast.

[CLOVIS]

I believe the build cache can safely be copied around, and the configuration cache cannot? Anyway I meant using a remote build cache, I should have been clearer.

[ephemient]

ah yeah remote build cache is definitely supported. but copying the whole .gradle folder is not meant to be, afaik

[ephemient]

well hmm. https://github.com/gradle/gradle-build-action#caching-build-state-between-jobs seems to include some build cache directories, so maybe that is supposed to be OK to reuse in some way. it's not documented as such in the previous link I sent, though

[CLOVIS]

I think it is, because the build cache is shared between multiple clones of the same project, so it must handle paths correctly. I know the documentation explicitly warns that it's unsafe with the configuration cache though.

[Nick Kleban]

To add context, I have:
• Gitlab with single runner.
• Cache stored locally on the runner.
• Build and configuration cache enabled.
• Gradle pre-installed on the image (no need for wrapper)

[Nick Kleban]

Is saving .gradle/caches/modules-2 will be enough to avoid dependency re-download?

And where is located configuration cache?

[CLOVIS]

Is saving .gradle/caches/modules-2 will be enough to avoid dependency re-download?

I think so, yes.

And where is located configuration cache?

In the repository's .gradle/configuration-cache directory. But do not share it between runs, it's not made for that and you risk it building the wrong stuff.

[Javier]

Why not? Recently the official Gradle GitHub Action brought back using the config cache.

[CLOVIS]

Did they? I have seen multiple times the Gradle team warn that this isn't supported. I don't know much more.

[Javier]

it was supported and it was removed due security concerns related to the secrets and they have fixed it with Gradle 8.6

[Javier]

but not about building wrong