:wave: Hello, I'm trying to debug remote cache mis...

# gradle

[wasyl]

👋 Hello, I'm trying to debug remote cache miss between two machines. I've set up toolchains and verified in a build scan that they're properly picked up. I then built our custom plugins and copied them between two machines (to rule out differences in how the plugins themselves are built). Now when I run the same tasks on both machines, I get the following differences in the caching keys (there's more output at the bottom with the same fingerprints and different one for the whole task again).
One machine is a Linux, java toolchain in build scan is Eclipse Temurin 17.0.6+10 (amd64). The other is an intel mac, build scan says toolchain is Eclipse Temurin 17.0.6+10 (x86_64)
Subsequent builds on the same machines produce the same keys as far as I can see.
The code to configure toolchains is:

kotlin {
    jvmToolchain {
        vendor.set(JvmVendorSpec.ADOPTIUM)
        languageVersion.set(JavaLanguageVersion.of(17))
    }
}

tasks.withType(KotlinCompile).configureEach {
    compilerOptions.jvmTarget.set(JvmTarget.JVM_11)
}

tasks.withType(JavaCompile).configureEach {
    options.release.set(Build.javaTargetVersion.majorVersion.toInteger())
}

I don't know what else could cause the differences and why the build cache between the two machines doesn't work. Unless it's not supposed to? This is all on Kotlin 1.8.10

[inaki]

Hi, some questions:
• Are you using version catalogs or project safe accessors?
• Root settings.gradle contains the project Name?

[inaki]

to be sure and if it's possible, can you remove the content of the Project folder ./gradle/GRADLE_VERSION and execute compare again?

[wasyl]

• we are using version catalogs and project safe accessors
• root settings.gradle does contain rootProject.name=...
• after removing ./.gradle the keys still differ (but are stable on both machines)

[wasyl]

I removed typesafe project accessors and I still see differences between the two machines. The cache keys without project accessors are also different than keys with accessors on the same machine

[tapchicoma]

I would advice you to run the build with debug logs on both machines and then investigate compiler arguments diff for the task under-the-question. In the logs you could search by Kotlin compiler args:

[wasyl]

Which flag/setting would I use? --debug?

[tapchicoma]

yes

[tapchicoma]

note that logs will be quite big so better to dump them into the file:

./gradlew build --debug > build.log

[inaki]

what called my attention from your screenshoot is that one of the differences is in the KotlinCompile_Decorated input.
In an scenario where you are using GE the differences looks like:

[inaki]

and we can focus in the classpatahSnapshotProeprties or compilerOptions but you can observe there is a difference in the task class too

[inaki]

this represents a "Task implementation is different"

[wasyl]

Yes, I didn’t know if difference in classes is expected or not, as it’s not clear from the logs if it’s part of the final cache key.
Anyway as far as I understand, the build should really use the same classes, unless amd vs intel makes a difference. I can’t think of anything that might cause the classes to differ, but our build is relatively complex.
We don’t have GE btw 😞

[wasyl]

Actually that particular project doesn’t even use our convention plugins, just

plugins {
    id("kotlin")
    id("java-test-fixtures")
    id("com.apollographql.apollo")
}

directly. That leaves either the build script or settings.gradle that might mess things up I think?

[inaki]

To discard any issue between platforms I would test one new simple project and compare the build cache keys for the KotlinCompiler tasks.

[inaki]

Last week I had a similar investigation and at the end was one stale state of the .gradle root folder

[wasyl]

You mean to compare cache keys for two projects on the same machine, for example the one above and a new simple one, with the same settings?

[inaki]

sorry I mean amd vs intel

[wasyl]

Last week I had a similar investigation and at the end was one stale state of the .gradle root folder

unfortunately during investigation I pretty much nuked everything and that didn’t help. Plus I observe the same cache keys on a second intel mbp, so at least that’s stable between the machines. Don’t have a second linux machine to check

[wasyl]

sorry I mean amd vs intel

so, a simple project on two platforms 👍 will do that

[inaki]

then, here an example of the same issue i was able to reproduce https://ge.solutions-team.gradle.com/c/acdplrem5t7tq/rizad2jjl2qxg/task-inputs

[mbonnin]

That leaves either the build script or settings.gradle that might mess things up I think?

I always wonder what goes into a given task cache key. My mental model is it's the full classpath?
So build.gradle, settings.gradle but also buildscript.classpath , plugins {} , project accessors, version catalog accessors and gradle model accessors

[mbonnin]

Potentially a lot of things

[wasyl]

I created https://github.com/lwasyl/kotlin-caching and I reproduce the exact same thing. Is it already something worth reporting to youtrack?

[wasyl]

Ah I forgot to set toolchains. But I happen to have the same jdks on both systems

[wasyl]

Filed https://youtrack.jetbrains.com/issue/KT-56929/Kotlin-tasks-not-cacheable-between-architectures
Something’s definitely off if such a simple project doesn’t work, and I verified caching between two intel macs works perfectly fine, so that rules out simple things like ~/.gradle/gradle.properties or init files

[tapchicoma]

@wasyl could you set default charset on amd machine to UTF-8 and again recheck caching?

[wasyl]

sure thing, that’d be ./gradlew assemble -Dfile.encoding=UTF-8 -Dorg.gradle.caching.debug=true?

[tapchicoma]

probably - make a build scan as well 🤔

[wasyl]

I tried that before and for some reason --scan was ignored with -Dorg.gradle.cahing.debug=true. I’ll try and make a scan separately too 👍

[inaki]

Forked the repo and tested.
8.0.1: https://ge.solutions-team.gradle.com/c/r77tmf6f3plo6/ugx5xkgns2rme/task-inputs
7.6: https://ge.solutions-team.gradle.com/c/6rcapxnybkaja/kivlrtkxrsxy2/task-inputs
going to check with the team too

[wasyl]

So it looks like a bug in Gradle?

I checked with -Dfile.encoding=UTF-8 and I still see the differences, let me know if you still want a scan/output.

[Alexander.Likhachev]

Just adding my two cents. I checked it on Kotlin compilation tasks, the result is successfully reused on switch between the same JDK but with different arch (Gradle 8.0.1, local build cache)

[wasyl]

Is it for the project I linked above?

[Alexander.Likhachev]

It’s not from your project, but I’ve just tested it on your project as well, and it works fine for me (on the same OS, but with the different arch of JDK)
So maybe OS has somehow became a part of cache key for JVM toolchains in Gradle

[wasyl]

I asked because I’m curious if your cache key for task :lib:compileKotlin from my project matches 8f5c7f6ed33f6dc43c6c2f1ec2cd9289 or c2706adf8bd6a5a2c45cd078ece23d57

[Alexander.Likhachev]

Screenshot 2023-02-23 at 21.20.56.png

[Alexander.Likhachev]

So, I see a different cache key on Intel Mac (and it’s different from x86-64 JDK on M1 Mac)

Summary:
The task has the same cache key on M1 Mac with aarch64 JDK and M1 Mac with x86-64 JDK (let’s call it A)
The task has a different cache key on Intel Mac with x86-64 JDK (let’s call it B)
A != B

[inaki]

@wasyl looks like more issues are showing up https://github.com/gradle/gradle/issues/24031 thank you for reporting this issue and hope to have a quick fix soon

[inaki]

tested with 8.0.2 in the same repo and works https://ge.solutions-team.gradle.com/c/3r2wdws4sddfc/res7obvygdkfs/task-inputs

[wasyl]

Yes, I also checked as soon as the commit fixing it was built, as I couldn't believe I had some 7.6 leftovers anywhere 😅 and I confirm this is now fixed 🙂