Does anyone use the Gradle configuration cache in a Kotlin native project? I can't get it to work in...

# gradle

[mbonnin]

Does anyone use the Gradle configuration cache in a Kotlin native project? I can't get it to work in CI. When starting from a clean state, "commonized" symbols like platform.* are not found 😞 . I can't really tell if it's a Gradle issue, Kotlin issue or just I'm holding it wrong? Does anyone have any success there?

[mbonnin]

Managed to reproduce in a minimal project, I filed https://youtrack.jetbrains.com/issue/KT-61359

[CLOVIS]

It seems to work fine on my sample project: https://gitlab.com/opensavvy/notes/kotlin-fullstack/-/tree/07879e40b829db1d3b17392331ae145eb5a88603

It's only a hello-world at the moment, so maybe issues arise later.

[CLOVIS]

I've had to disable it in later versions because of Dokka, but haven't noticed issues with the Kotlin plugins themselves.

[mbonnin]

Do you have a CI in that project?

[mbonnin]

Ah yes, found it! (sorry, not too familiar with gitlab)

[CLOVIS]

Yes, it just runs ./gradlew check IIRC

[CLOVIS]

You can see the latest trace here: https://gitlab.com/opensavvy/notes/kotlin-fullstack/-/jobs/4875738685

[CLOVIS]

I wonder if you somehow reuse the confirmation cache between executions? AFAIK that's not supported by Gradle itself (so, at the moment, the configuration cache is useless in CI if you clean your environment)

[mbonnin]

I can actually reproduce on my machine locally as well. Seems to be a race condition in how the native compiler is setup

[mbonnin]

I can't reproduce with your project though, maybe because it's a darwin/macos thing 🤔 (looks like you only have 1 linux target?)

[CLOVIS]

I only support LinuxX64, but AFAIK you should be able to add any native target, I just depend on Kotter which supports MacOS

I haven't configured it in the project because I don't have a device to run it anyway

[Oleg Yukhnevich]

the configuration cache is useless in CI

not entirely 🙂
Im using CC in my projects on CI, to be able to run test tasks in single project in parallel. F.e. if we have KMP module with N kotlin targets, we can run tests for those targets in parallel (on GitHub actions Linux runners has 2 cores and MacOS runners has 3 cores!)
And it definitely reduces tests execution times

[mbonnin]

@Oleg Yukhnevich so I infer you're using the CC successfully in CI then?

[Oleg Yukhnevich]

BTW, at my side (https://github.com/whyoleg/cryptography-kotlin/actions/runs/5798251918) I don’t see similar issues with platform symbols - even with custom cinterop

[Oleg Yukhnevich]

Note: there are some failed steps there because of other issues, but check task works

[mbonnin]

Do you cache .konan in your GitHub Actions cache by any chance?

[Oleg Yukhnevich]

Yeah, I do.

[Oleg Yukhnevich]

I can try to reset cache and rerun this pipeline with Configuration Cache 🙂

[Oleg Yukhnevich]

clicked rerun - https://github.com/whyoleg/cryptography-kotlin/actions/runs/5798251918 (could you see progress?)
P.S. it was easier then expected, I haven’t committed for some time and caches expired and were deleted automatically 🙃

[mbonnin]

Yep, it's all running 👌

[mbonnin]

I can reproduce locally though: https://gist.github.com/martinbonnin/001f47fa4bd1a994ac6876fcfe74df40

[mbonnin]

GetRandom.kt:7:40 [UNRESOLVED_REFERENCE] Unresolved reference: internal

[Oleg Yukhnevich]

sad 😞
Then Im also affected 🙂
Though, for now I mostly need CC on CI, as locally everything works more or less ok, and Im mostly waiting for linking native binaries and not gradle configuration…

[Oleg Yukhnevich]

You know what, I missed something: this workflow has cache-disabled: true, so there were no cache at all
So looks like the problem is in assemble task, cause check task works on CI

[mbonnin]

I'm pretty sure it's with intermediate source sets and compile${Foo}KotlinMetadata

[mbonnin]

I never know what check depends on. If it's not publishing (it shouldn't) then maybe it doesn't need the KotlinMetadata parts

[CLOVIS]

check should depend on anything and everything that lets you decide in an automated way whether your code is good or not (unit tests, integration tests, coverage checks, quality gates, everything)

[Oleg Yukhnevich]

yeah, it doesn’t compile intermediate source sets
but, also in your gist I see that random:compileKotlinLinuxX64 failed, but on CI run it was executed successfully

[mbonnin]

It's :cryptography-random:compileAppleMainKotlinMetadata that failed

[Oleg Yukhnevich]

there is also in your gist:

> Task :cryptography-random:compileKotlinLinuxX64 FAILED
e: file:///Users/mbonnin/git/cryptography-kotlin/cryptography-random/src/linuxMain/kotlin/GetRandom.kt:7:40 [UNRESOLVED_REFERENCE] Unresolved reference: internal
e: file:///Users/mbonnin/git/cryptography-kotlin/cryptography-random/src/linuxMain/kotlin/GetRandom.kt:22:60 [UNRESOLVED_REFERENCE] Unresolved reference: GRND_NONBLOCK

[mbonnin]

Ah yes! double failure 🤔

[Oleg Yukhnevich]

it’s declaration coming from cinterop

[mbonnin]

Weird

[Oleg Yukhnevich]

I see, that there are slightly different tasks were executed, I think, that because of CC some dependency on task is missing somehow
I think, that to understand what’s going on we need to run 2 builds:
1. ./gradlew assemble --continue
2. ./gradlew assemble --continue --configuration-cache
save log / which tasks were executed in each run. Between runs delete .konan folder
may be --dry-run also will be enough here to see task list, but not sure
then compare them and pass info in YouTrack issue 🙂

[mbonnin]

Let me try that!

[Oleg Yukhnevich]

just in case, why I say like this: I see, that even first task that was executed in your gist is directly cryptography-random:compileKotlinLinuxX64 - but it could not be true, as it should depends at least on cryptography-random:cinteropRandomLinuxX64 - which should generate cinterop declarations

[mbonnin]

so tasks dependencies would be different under CC?

[Oleg Yukhnevich]

could be, hard to know at my side, Im not fully an expert on how CC works under the hood
Also I don’t fully know how CC is implemented in KGP. There were a lot of improvements there in latest versions but I still see some issues. + last time I look there, there were some complex logic to support cinterop and commonizer there

[mbonnin]

Agreed, that stuff is hard. At least we have a reproducer so hopefully the Kotlin team can investigate this 🤞