kotlin 1.5.21 i observe remote cache misses for any kotlinCompile task between macOs and Linux with ...

# gradle

[dsvoronin]

kotlin 1.5.21
i observe remote cache misses for any kotlinCompile task between macOs and Linux
with single difference:
Appending input value fingerprint for 'kotlinJavaToolchainProvider' to build cache key: 579ef60211f6a32e39cd244937049220
Appending input value fingerprint for 'kotlinJavaToolchainProvider' to build cache key: 7b38054c10085dfb665417dd82eb8269
why it could be different?
nested kotlinJavaToolchainProvider.javaVersion are equal
Why it's even calculated as a part of compilation task cache if only javaVersion property marked as @Input ?

[dsvoronin]

tried to set jvm on both machines to single dir, like /opt/jdk - didn't help

[tapchicoma]

are you using different JDK versions locally and on CI? 🤔

[dsvoronin]

nope, it's the same

[dsvoronin]

i also tried to install same minor versions and same vendor

[tapchicoma]

this should not affect it, only major version is used

[tapchicoma]

is it possible to try 1.5.30 version?

[dsvoronin]

yep, i already tried it, but with the same result
only thing i'm not sure ... i use kotlin dsl and it's bundled with 1.5.21 version (gradle 7.2)

[dsvoronin]

so i see kotlin compiler daemon started with 1.5.21 in classpath for scripts compilation before 1.5.30 started for project kotlin tasks

[dsvoronin]

via VisualVm

[tapchicoma]

1.5.2x version has already some parts of toolchain support, but it should not affect remote build cache misses.

Also we have tests for toolchain cache relocation and they are green 🤔

Would be nice if you could provide some repro project

[dsvoronin]

sure, i also did relocation tests for single machine (macos) with different project root dirs - and it's OK

[dsvoronin]

problem started between OSes, which is odd

[dsvoronin]

i don't understand how gradle treats this field
https://github.com/JetBrains/kotlin/blob/7aa882b921eeddcc9144f2c6de46fa2e91cdb4d3/[…]in/org/jetbrains/kotlin/gradle/tasks/UsesKotlinJavaToolchain.kt

it's nested and with single input javaVersion field
so if javaVersions are equals, kotlinJavaToolchainProvider should be too? right? it's not

[tapchicoma]

I would expect only javaVersion should be taken into consideration of this @Nested input

[dsvoronin]

me too, but that's my debug cache output for kotlin compile: https://pastebin.com/rzATTSJS

[tapchicoma]

I will ask someone from Gradle

[dsvoronin]

thanks! i'll do some digging too;
i want at least to workaround this for now for our project; but it seems impossible without knowing how this nested fields works regarding caching

we warms our cache on CI only from linux machines, but consumes it for a lot of MacOSes
things goes pretty well until recent. i suppose ~1.5 kotlin update

[tapchicoma]

btw which Gradle version are you using?

[dsvoronin]

Gradle 7.2
AGP 7.0.2
kotlin 1.5.21

[tapchicoma]

From @Nested documentation:

The implementation of the nested bean is tracked as an input, too. This allows tracking behavior such as org.gradle.api.Actions as task inputs.

[tapchicoma]

though implementation is the same for Linux and MacOS 🤔

[dsvoronin]

thanks, i read that, but don't understand how is "nested" hash calculated.
what is taken into account?

[dsvoronin]

time to read sources i think)

[tapchicoma]

I greatly suspect currentJvm property causes difference in hashes. Though I need to also figure out how hash for nested object is calculated

[dsvoronin]

but it's explicitly marked as '@Internal'? or it's 1.5.30+ change?

[dsvoronin]

i see

[tapchicoma]

yes, it is marked as @Internal in both 1.5.20 and 1.5.30 releases

[tapchicoma]

have you tested Linux CI/Local Linux build cache hit?

[dsvoronin]

yes, it's working fine

[no]

Can you print the value out for both builds?

[tapchicoma]

@wolf this one

[wolf]

My current guess is that the plugin classloader somehow contains different classes between the two builds. Do have have buildSrc or included builds for the plugins you use in your build? I’d expect that the classes compiled from build src are different on the different operating systems.

[wolf]

This may be caused by using a different JDK for running the build.

[tapchicoma]

I thought that JDK version should not affect the build itself, just the JVM tasks 🤔

[wolf]

It affects the version used to build things in buildSrc, if you don’t use toolchains in buildSrc. And there the classes could be different.

[tapchicoma]

@dsvoronin could you also try to downgrade Gradle to 7.1 version and check if it is still happenning?

[dsvoronin]

Can you print the value out for both builds?

values are in first message? or i didn't understand you

[dsvoronin]

Do have have buildSrc  or included builds for the plugins you use in your build? I’d expect that the classes compiled from build src are different on the different operating system

yes, i'm using included builds like here
https://github.com/avito-tech/avito-android

projects are about identical in terms of structure

[dsvoronin]

could you also try to downgrade Gradle to 7.1 version and check if it is still happenning?

yep, let me try this one

[dsvoronin]

It affects the version used to build things in buildSrc, if you don’t use toolchains in buildSrc. And there the classes could be different.

is there any way to workaround this before kotlin 1.5.30 with proper toolchains support?

[dsvoronin]

MacOs compile task debug cache: https://pastebin.com/TQCxbR3Q
Linux compile task debug cache:
https://pastebin.com/mix5tBRf

[dsvoronin]

that's full cache debug for builds on both machines, on same versions as before

[dsvoronin]

i also observe:

Appending implementation to build cache key: org.jetbrains.kotlin.gradle.tasks.KotlinCompileWithWorkers_Decorated@76e72c2af8253c6fd36ebffa9b234b23
Appending additional implementation to build cache key: org.jetbrains.kotlin.gradle.tasks.KotlinCompileWithWorkers_Decorated@76e72c2af8253c6fd36ebffa9b234b23
Appending additional implementation to build cache key: org.jetbrains.kotlin.gradle.tasks.AbstractKotlinCompileTool$1@76e72c2af8253c6fd36ebffa9b234b23

different values on top of the files, but it's unclear if its affecting resulting hash

[wolf]

Seems like the classloader hashes for the Kotlin plugin classpath change:

org.jetbrains.kotlin.gradle.tasks.KotlinCompileWithWorkers_Decorated@76e72c2af8253c6fd36ebffa9b234b23
org.jetbrains.kotlin.gradle.tasks.KotlinCompileWithWorkers_Decorated@ebd5c7dbc036652a668e7dbe1c302a9a

[wolf]

That means that the plugin classloader is different.

[dsvoronin]

and it's also affecting the kotlinJavaToolchainProvider property?

[tapchicoma]

could be. 1.5.21 has different toolchain provider implementation then 1.5.30

[dsvoronin]

ok, is there any way to align this plugin classloaders?
that's the same commit and same sources, only difference is OS

[dsvoronin]

i also tried to align jdks on these machines, checking: absolute path / version (major/minor//patch) and vendor

[wolf]

The absolute path does not matter. The version and vendor do.

[dsvoronin]

@tapchicoma seems like downgrading to 7.1.x will be a challenging one; because build already uses some new things, like new declarations in version catalog toml

[tapchicoma]

in kind of similar case downgrade helped

[dsvoronin]

i could try to use nightly/dev builds of 7.3, but seems like 1.5.30 still not merged
https://github.com/gradle/gradle/pull/18148

[tapchicoma]

maybe you could rollback commits up-to 7.2 upgrade and check kotlin upgrade from there?

[tapchicoma]

7.3-nightly from our tests will not help

[dsvoronin]

yep, my second option will be to also warmup cache on mac machine in pair with linux one

[dsvoronin]

not sure if it will help

[wolf]

I can’t see how downgrading the Gradle version would help here.

[dsvoronin]

@wolf Is there any guide on how to debug and fix my plugins classloader problem?

[no]

can you apply this gist to your build.gradle file and send us a build scan? https://gist.github.com/ghale/324225854672e85131da0aa73857c1ce

if it isn't possible to send a build scan, replace the buildScan.value lines with println statements

[dsvoronin]

yep, will do in couple of hours, thanks!

[dsvoronin]

https://gist.github.com/dsvoronin/c26db48a1e56f530462f0335af4cb8dc
Hope i did all of that correctly

[wolf]

I don’t see the Fingerprints: output in the diff. Does this mean they are the same?

[wolf]

If they are, it seems that somehow the classloader order is different between the two build: The Linux build has conventions.jar first and then a parent classloader with only dependencies.jar . Where do those two JARs come from?

[wolf]

Is that build-logic-settings:dependencies and build-logic:conventions?

[dsvoronin]

Is that build-logic-settings:dependencies and build-logic:conventions?

yes

[dsvoronin]

I don’t see the Fingerprints: output in the diff. Does this mean they are the same?

That's strange, because i used snipper from gist; i'll double check

[dsvoronin]

fingerprints are available in build scan
but i'll do a log update; somehow it was not printed

[dsvoronin]

ah ok, gist truncates log files because it's exceeds its limit;
so if build scan is enough - i'll just remove logs

[dsvoronin]

Problem was in out project's flag: syncAllApps, which is disabled locally to prevent synchronization of all apps in monorepo.
What i'm not expected is project dependency accessors in module compile kotlin classpath.
Why is that working that way? In my opinion it shouldn't affect module compilation at all

https://gist.github.com/dsvoronin/c26db48a1e56f530462f0335af4cb8dc#file-difference-md

[dsvoronin]

and btw it wasn't between Mac and Linux at all, it was a false assumption, because i work on linux and cache was ok for me, and ci that pushing cache is only linux machine,
but local developers machines are mostly MacOs with disabled flag

[no]

Nice find @dsvoronin! Could you clarify what syncAllApps does? I have a hard time understanding. Does it selectively add/remove projects in settings.gradle?

[dsvoronin]

yep, you got it right, there are multiple app-modules in a single gradle project that is conditionally enables

[dsvoronin]

some dev’s working on one and two, some only on third app

[dsvoronin]

ci only warmed “all enabled” scenario, which was rare (only me basically)

[no]

What's the benefit to conditionally enabling / disabling these apps?
Without a consistent input to the dependency resulution strategy, the order of the classpath cannot be deterministic.

[dsvoronin]

benefit is in IDE responsiveness

[dsvoronin]

measures are not up to date, but about year or two ago enabling only one app gave significant impact on essential ide functions like syntax analysis, code compltion etc.

[no]

Makes sense.