Does anyone has any issues with linking native bin...

# eap

[Oleg Yukhnevich]

Does anyone has any issues with linking native binaries on linux/macos with kotlin 1.8.20-Beta?
like:
• https://github.com/whyoleg/cryptography-kotlin/actions/runs/4151887365/jobs/7182476137:

e: /home/runner/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-coroutines-test-linuxx64/1.6.4/4757f4ccaeca3c186c03adb80dd4125932850259/kotlinx-coroutines-test is cached (in /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/org.jetbrains.kotlinx:kotlinx-coroutines-test/unspecified/1e60851fbe3383bc94735442f181a9273d1e96adaab6fe4b95811738dc5e378e/org.jetbrains.kotlinx:kotlinx-coroutines-test-cache/bin/liborg.jetbrains.kotlinx:kotlinx-coroutines-test-cache.a), but its dependency isn't: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/common/stdlib

• https://github.com/rsocket/rsocket-kotlin/actions/runs/4152472478/jobs/7183450649

e: /home/runner/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/atomicfu-linuxx64/0.19.0/e5e28a197f697b77d49846bfec2fce6e0847e5ed/atomicfu is cached (in /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/org.jetbrains.kotlinx:atomicfu/unspecified/7ec08e59096adf58ceb23075044b5b12968ce418b16d8d23364485d49f5fe48c/org.jetbrains.kotlinx:atomicfu-cache/bin/liborg.jetbrains.kotlinx:atomicfu-cache.a), but its dependency isn't: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/common/stdlib

or

e: /home/runner/.gradle/caches/modules-2/files-2.1/io.ktor/ktor-utils-linuxx64/2.2.3/1f96940f71493794b49a4fe22d25e448ae663854/ktor-utils is cached (in /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/io.ktor:ktor-utils/unspecified/cd659155ccdeccb8b955ac996626c4018e6b1b19d85023c63b5d72de04e3affe/io.ktor:ktor-utils-cache/bin/libio.ktor:ktor-utils-cache.a), but its dependency isn't: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/common/stdlib

[Adam S]

could you try adding --no-parallel to the Gradle command? Kotlin/Native doesn’t follow the Gradle API so it’s a bit janky


https://kotlinlang.slack.com/archives/C3SGXARS6/p1585734327100500

https://youtrack.jetbrains.com/issue/KT-56309

[Oleg Yukhnevich]

looks like it’s not an issue with parallel https://github.com/whyoleg/cryptography-kotlin/actions/runs/4155528407/jobs/7188679896
but disabling native caches helped in this case

[Adam S]

it might help to cache Konan separately from Gradle, with a cache key that’s based on the OS and Kotlin version. Or manually delete the caches every time there’s a Kotlin version change.

[Adam S]

btw have you thought about adding concurrency cancellation? I saw you had a few jobs waiting for old jobs to finish https://docs.github.com/en/actions/using-jobs/using-concurrency

just add

concurrency:
  group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.head_ref || github.ref }}'

(copied from https://docs.github.com/en/actions/examples/using-concurrency-expressions-and-a-test-matrix)

[Oleg Yukhnevich]

by native caches, I mean using this flag in gradle properties kotlin.native.cacheKind=none
And BTW, I’ve already disabled gradle caches for kotlin-1.8.20 branch at all, so it should not be an issue with outdated caches

[Adam S]

ahh right, I didn’t know about that flag. So many caches!

[Oleg Yukhnevich]

it might help to cache Konan separately from Gradle

yeah, thought, it’s harder to do it correctly, because then I need to know Kotlin version inside GA workflow, but I will think about it, thx!

[Adam S]

I cache .konan/ without the Kotlin version, just based on the OS, and trust that eventually GitHub will delete the cache once it gets too big.

[jb.igor.chevdar]

There is also new flag: kotlin.native.cacheOrchestration=gradle (we’ve reworked caches in 1.8.20 and this flag is a way to fallback to the previous implementation).

[jb.igor.chevdar]

@Oleg Yukhnevich Does the bug reproduce with clean build? (And if it does, is it persistent?)

[Oleg Yukhnevich]

yeah, it’s reproduced every time on branch https://github.com/whyoleg/cryptography-kotlin/tree/kotlin-1.8.20
Specifically after this commit https://github.com/whyoleg/cryptography-kotlin/commit/a2de373a191edf68ab4333de481130c8dda1e90b it was reproduced in 5 subsequent builds:
• with/without gradle cache (with and without clean build)
• with/without gradle parallel flag
BTW, locally on Mac M1 it’s not reproduced at my side (don’t have x64 mac or linux at my side to test locally)

[jb.igor.chevdar]

BTW, locally on Mac M1 it’s not reproduced at my side

This is expected (the caches are turned off now on Mac M1)

[Oleg Yukhnevich]

@jb.igor.chevdar adding kotlin.native.cacheOrchestration=gradle instead of kotlin.native.cacheKind=none also solves linkage error

[Oleg Yukhnevich]

BTW, should I create an issue in YT with this info to keep tracking it? :)

[jb.igor.chevdar]

Yeah, this would be great!

[Oleg Yukhnevich]

And is there an issue regarding native cache support for m1?

[jb.igor.chevdar]

AFAIK, there’s not.

[Oleg Yukhnevich]

Then I will create it :)

[Oleg Yukhnevich]

Created issues:
• for broken caches: https://youtrack.jetbrains.com/issue/KT-56611/Native-new-native-caches-are-broken
• for M1 support https://youtrack.jetbrains.com/issue/KT-56610/Native-caches-are-not-available-on-Mac-M1

[jb.igor.chevdar]

I’m trying to reproduce the bug locally but no luck so far:
I’ve cloned https://github.com/whyoleg/cryptography-kotlin.git, checked out a2de373 (just after switching to 1.8.20-Beta) and run ./gradlew clean linkDebugTestMacosX64.
I’m not getting any errors and the logs show that all the caches are being built successfully 🤷

[Oleg Yukhnevich]

try to run check -PskipTest may be ? (it’s what is executed on CI)

[jb.igor.chevdar]

try to run check -PskipTest

Nope, still no error.

[jb.igor.chevdar]

Could you add this snippet to build.gradle.kts for appropriate projects:

tasks.withType<org.jetbrains.kotlin.gradle.tasks.KotlinNativeLink>().configureEach {
    toolOptions.verbose.set(true)
}

(I’ve added to cryptography-tests/behavior-tests/build.gradle.kts, cryptography-tests/compatibility-tests/build.gradle.kts and cryptography-tests/test-utils/build.gradle.kts) and rerun CI? (This will add some additional cache-related logging and, hopefully, will shed some light to the problem).

[Oleg Yukhnevich]

I can add it

But, first, I’ve found, that in another project (rsocket-kotlin) there is the same issue for basic build, but no in another workflow (with kotlin matrix testing)
And the only difference there, is that in workflow, where issue happened, there is a command, which change KONAN_DATA_DIR (echo "KONAN_DATA_DIR=${HOME}/.gradle/konan" >> $GITHUB_ENV)- cryptograhy-kotlin also do this (to cache konan distribution by Github Actions)
Now I dropped it - and build was successful with enabled native caches

[Oleg Yukhnevich]

Here are logs with verbose flag set for linux

:cryptography-tests:cryptography-behavior-tests:linkDebugTestLinuxX64 FAILED	
v: Using Kotlin home directory dist/kotlinc	
v: CACHING /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/common/stdlib	
v:     dependencies:	
        	
v:     caches used:	
        	
v:     cache dir: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC	
v: CACHING /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/platform/linux_x64/org.jetbrains.kotlin.native.platform.posix	
v:     dependencies:	
        	
v:     caches used:	
        /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/common/stdlib: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/stdlib-cache	
v:     cache dir: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC	
v: CACHING /home/runner/.gradle/caches/modules-2/files-2.1/dev.whyoleg.kcwrapper/libcrypto3-api-linuxx64/0.1.0-SNAPSHOT/5374a360660b3a41958d8f69ed658938c39801d5/libcrypto3-api-linuxx64-0.1.0-SNAPSHOT-cinterop-api	
v:     dependencies:	
        	
v:     caches used:	
        /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/common/stdlib: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/stdlib-cache	
        /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/platform/linux_x64/org.jetbrains.kotlin.native.platform.posix: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/org.jetbrains.kotlin.native.platform.posix-cache	
v:     cache dir: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/dev.whyoleg.kcwrapper:libcrypto3-api-cinterop-api/unspecified/a77298f2dded0352a4cfa04c491d1bdf75d452c7dfd9994244d86e7ece4b3fa8	
v: CACHING /home/runner/.gradle/caches/modules-2/files-2.1/dev.whyoleg.kcwrapper/libcrypto3-api-linuxx64/0.1.0-SNAPSHOT/20575bea5a23dcb98d4162c3f0a4c1798383053e/libcrypto3-api-linuxx64-0.1.0-SNAPSHOT	
v:     dependencies:	
        /home/runner/.gradle/caches/modules-2/files-2.1/dev.whyoleg.kcwrapper/libcrypto3-api-linuxx64/0.1.0-SNAPSHOT/5374a360660b3a41958d8f69ed658938c39801d5/libcrypto3-api-linuxx64-0.1.0-SNAPSHOT-cinterop-api.klib	
v:     caches used:	
        /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/common/stdlib: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/stdlib-cache	
        /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/platform/linux_x64/org.jetbrains.kotlin.native.platform.posix: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/org.jetbrains.kotlin.native.platform.posix-cache	
        /home/runner/.gradle/caches/modules-2/files-2.1/dev.whyoleg.kcwrapper/libcrypto3-api-linuxx64/0.1.0-SNAPSHOT/5374a360660b3a41958d8f69ed658938c39801d5/libcrypto3-api-linuxx64-0.1.0-SNAPSHOT-cinterop-api.klib: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/dev.whyoleg.kcwrapper:libcrypto3-api-cinterop-api/unspecified/a77298f2dded0352a4cfa04c491d1bdf75d452c7dfd9994244d86e7ece4b3fa8/dev.whyoleg.kcwrapper:libcrypto3-api-cinterop-api-cache	
v:     cache dir: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/dev.whyoleg.kcwrapper:libcrypto3-api/unspecified/53f470a7e573191653f35b1e7cdba39d11612d6641ba5cda5a1f308286a6596e	
v: CACHING /home/runner/.gradle/caches/modules-2/files-2.1/dev.whyoleg.kcwrapper/libcrypto3-static-linuxx64/0.1.0-SNAPSHOT/faf10573368e41a45a8e57be717051b0e77ca488/libcrypto3-static-linuxx64-0.1.0-SNAPSHOT-cinterop-static	
v:     dependencies:	
        	
v:     caches used:	
        /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/common/stdlib: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/stdlib-cache	
v:     cache dir: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/dev.whyoleg.kcwrapper:libcrypto3-static-cinterop-static/unspecified/4db3c4dd516de69c49b9e523331a546d1a5dfba2e0e88fed4d2527ff9b3256c9	
v: CACHING /home/runner/.gradle/caches/modules-2/files-2.1/dev.whyoleg.kcwrapper/libcrypto3-static-linuxx64/0.1.0-SNAPSHOT/36198a4ae7cc71380b5d50b25ecd76e9ee28a1b9/libcrypto3-static-linuxx64-0.1.0-SNAPSHOT	
v:     dependencies:	
        /home/runner/.gradle/caches/modules-2/files-2.1/dev.whyoleg.kcwrapper/libcrypto3-static-linuxx64/0.1.0-SNAPSHOT/faf10573368e41a45a8e57be717051b0e77ca488/libcrypto3-static-linuxx64-0.1.0-SNAPSHOT-cinterop-static.klib	
        /home/runner/.gradle/caches/modules-2/files-2.1/dev.whyoleg.kcwrapper/libcrypto3-api-linuxx64/0.1.0-SNAPSHOT/5374a360660b3a41958d8f69ed658938c39801d5/libcrypto3-api-linuxx64-0.1.0-SNAPSHOT-cinterop-api.klib	
        /home/runner/.gradle/caches/modules-2/files-2.1/dev.whyoleg.kcwrapper/libcrypto3-api-linuxx64/0.1.0-SNAPSHOT/20575bea5a23dcb98d4162c3f0a4c1798383053e/libcrypto3-api-linuxx64-0.1.0-SNAPSHOT.klib	
v:     caches used:	
        /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/common/stdlib: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/stdlib-cache	
        /home/runner/.gradle/caches/modules-2/files-2.1/dev.whyoleg.kcwrapper/libcrypto3-static-linuxx64/0.1.0-SNAPSHOT/faf10573368e41a45a8e57be717051b0e77ca488/libcrypto3-static-linuxx64-0.1.0-SNAPSHOT-cinterop-static.klib: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/dev.whyoleg.kcwrapper:libcrypto3-static-cinterop-static/unspecified/4db3c4dd516de69c49b9e523331a546d1a5dfba2e0e88fed4d2527ff9b3256c9/dev.whyoleg.kcwrapper:libcrypto3-static-cinterop-static-cache	
        /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/platform/linux_x64/org.jetbrains.kotlin.native.platform.posix: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/org.jetbrains.kotlin.native.platform.posix-cache	
        /home/runner/.gradle/caches/modules-2/files-2.1/dev.whyoleg.kcwrapper/libcrypto3-api-linuxx64/0.1.0-SNAPSHOT/5374a360660b3a41958d8f69ed658938c39801d5/libcrypto3-api-linuxx64-0.1.0-SNAPSHOT-cinterop-api.klib: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/dev.whyoleg.kcwrapper:libcrypto3-api-cinterop-api/unspecified/a77298f2dded0352a4cfa04c491d1bdf75d452c7dfd9994244d86e7ece4b3fa8/dev.whyoleg.kcwrapper:libcrypto3-api-cinterop-api-cache	
        /home/runner/.gradle/caches/modules-2/files-2.1/dev.whyoleg.kcwrapper/libcrypto3-api-linuxx64/0.1.0-SNAPSHOT/20575bea5a23dcb98d4162c3f0a4c1798383053e/libcrypto3-api-linuxx64-0.1.0-SNAPSHOT.klib: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/dev.whyoleg.kcwrapper:libcrypto3-api/unspecified/53f470a7e573191653f35b1e7cdba39d11612d6641ba5cda5a1f308286a6596e/dev.whyoleg.kcwrapper:libcrypto3-api-cache	
v:     cache dir: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/dev.whyoleg.kcwrapper:libcrypto3-static/unspecified/5e68dc5540b1660f048da55ec1fa2d3914742772b4521b264188a8dd22553798	
v: CACHING /home/runner/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/atomicfu-linuxx64/0.17.3/dbf7ab263d8ddbeedc0376daf2646be87267276b/atomicfu-cinterop-interop	
v:     dependencies:	
        	
v:     caches used:	
        /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/common/stdlib: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/stdlib-cache	
        /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/platform/linux_x64/org.jetbrains.kotlin.native.platform.posix: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/org.jetbrains.kotlin.native.platform.posix-cache	
v:     cache dir: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/org.jetbrains.kotlinx:atomicfu-cinterop-interop/unspecified/42b5aaaa9509966a6cecd32eb892f4ea93198b459e16daafb7d1c91502f6b845	
v: CACHING /home/runner/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/atomicfu-linuxx64/0.17.3/ea2603c4343b2f1e779053662a6e2879842af2d9/atomicfu	
w: Opt-in requirement marker kotlinx.coroutines.ExperimentalCoroutinesApi is unresolved. Please make sure it's present in the module dependencies	
w: Opt-in requirement marker dev.whyoleg.cryptography.provider.CryptographyProviderApi is unresolved. Please make sure it's present in the module dependencies	
w: Opt-in requirement marker dev.whyoleg.cryptography.algorithms.InsecureAlgorithm is unresolved. Please make sure it's present in the module dependencies	
v:     dependencies:	
        /home/runner/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/atomicfu-linuxx64/0.17.3/dbf7ab263d8ddbeedc0376daf2646be87267276b/atomicfu-cinterop-interop.klib	
v:     caches used:	
        /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/common/stdlib: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/stdlib-cache	
        /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/platform/linux_x64/org.jetbrains.kotlin.native.platform.posix: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/org.jetbrains.kotlin.native.platform.posix-cache	
        /home/runner/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/atomicfu-linuxx64/0.17.3/dbf7ab263d8ddbeedc0376daf2646be87267276b/atomicfu-cinterop-interop.klib: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/org.jetbrains.kotlinx:atomicfu-cinterop-interop/unspecified/42b5aaaa9509966a6cecd32eb892f4ea93198b459e16daafb7d1c91502f6b845/org.jetbrains.kotlinx:atomicfu-cinterop-interop-cache	
v:     cache dir: /home/runner/.gradle/konan/kotlin-native-prebuilt-linux-x86_64-1.8.20-Beta/klib/cache/linux_x64-gSTATIC/org.jetbrains.kotlinx:atomicfu/unspecified/ebd8a9343ad47170e86dba19edc443e3800fc95e5cb9ea396362e2a8d506b0a0

[jb.igor.chevdar]

Did the above build end with the error?

[Oleg Yukhnevich]

With verbose flag and konan dir changed - yes
I can also run one more build with no changing konan dir + verbose flag

[jb.igor.chevdar]

Ok, I’ve managed to reproduce the bug locally: you have to override KONAN_DATA_DIR (as you hinted) but not to anywhere but rather to something exactly inside ~/.gradle directory; looks like there’s some kind of names clash here, I will investigate.

[Oleg Yukhnevich]

Magic coincidence 😄