Cross-posting

# eap

[aurimas]

Cross-posting

[anton.lakotka]

It might be related to these:
• KT-81724/IntelliJ-successfully-resolves-imports-that-are-not-actually-available-in-the-given-module
• KT-83917/compileCommonMainKotlinMetadata-fails-in-CMP-core-repository-with-Kotlin-2.3.20-Beta1
I'll respond with details later.

[aurimas]

any suggestions on how to verify if it is one of these issues?

[anton.lakotka]

Sure,

Lets try running this:

./gradlew :XXX:transformCommonMainDependenciesMetadata --rerun --debug  |  grep GranularMetadataTransformation

it should print this for savedstate

Transformation result of dependency androidx.savedstate:savedstate:1.4.0: choose, dependency = androidx.savedstate:savedstate:1.4.0, sourceSets = [*commonMain]

After you can also check build dir

build/kotlinTransformedMetadataLibraries/commonMain/

there should be commonMain.json file containing list of artifacts for common classpath.

And then you can also check how this dependency is resolved for each target + metadata:

:dependencyInsight --dependency androidx.savedstate:savedstate:1.4.0 --configuration commonMainResolvableDependenciesMetadata
:dependencyInsight --dependency androidx.savedstate:savedstate:1.4.0 --configuration jvmCompileClasspath (most likely for you it will be desktopCompileClasspath)
:dependencyInsight --dependency androidx.savedstate:savedstate:1.4.0 --configuration jsCompileClasspath
:dependencyInsight --dependency androidx.savedstate:savedstate:1.4.0 --configuration wasmJsCompileClasspath
:dependencyInsight --dependency androidx.savedstate:savedstate:1.4.0 --configuration linuxArm64CompileKlibraries
... etc

I've checked androidx.savedstate:savedstate:1.4.0 on my local project and it seems working OK

If you don't mind debugging, you can debug this method org.jetbrains.kotlin.gradle.plugin.mpp.SourceSetVisibilityProvider#getVisibleSourceSets when running transformCommonMainDependenciesMetadata task.
Or just give me reproducer.

I'm not entirely sure what went wrong for you.

[aurimas]

I'll give this a try, thanks for the pointers

[anton.lakotka]

Did you find out what the problem was?

[aurimas]

AndroidX build practically never finishes with --debug as we produce so much log output

Looking the build directory

$ ls -lsa ../../out/androidx/navigation/navigation-common/build/kotlinTransformedMetadataLibraries/commonMain/
total 296
  4 drwxr-xr-x 2 aurimas primarygroup   4096 Feb 24 15:20 .
  4 drwxr-xr-x 3 aurimas primarygroup   4096 Feb 24 15:20 ..
  4 -rw-r--r-- 1 aurimas primarygroup   4022 Feb 24 15:20 androidx.annotation-annotation-1.9.1-commonMain-WmoUwA.klib
 36 -rw-r--r-- 1 aurimas primarygroup  33982 Feb 24 15:20 androidx.collection-collection-1.5.0-commonMain-j7f1lg.klib
  4 -rw-r--r-- 1 aurimas primarygroup   1348 Feb 24 15:20 commonMain.json
192 -rw-r--r-- 1 aurimas primarygroup 194815 Feb 24 15:20 org.jetbrains.kotlin-kotlin-stdlib-2.1.20-commonMain-uBNgdg.klib
 52 -rw-r--r-- 1 aurimas primarygroup  50160 Feb 24 15:20 org.jetbrains.kotlinx-kotlinx-serialization-core-1.7.3-commonMain-s2qClw.klib

commonMain.json

[
  {
    "moduleId": "module androidx.annotation:annotation:1.9.1",
    "file": "/usr/local/google/home/aurimas/Code/androidx-main/out/androidx/navigation/navigation-common/build/kotlinTransformedMetadataLibraries/commonMain/androidx.annotation-annotation-1.9.1-commonMain-WmoUwA.klib",
    "sourceSetName": "commonMain"
  },
  {
    "moduleId": "module org.jetbrains.kotlin:kotlin-stdlib:2.1.20",
    "file": "/usr/local/google/home/aurimas/Code/androidx-main/out/androidx/navigation/navigation-common/build/kotlinTransformedMetadataLibraries/commonMain/org.jetbrains.kotlin-kotlin-stdlib-2.1.20-commonMain-uBNgdg.klib",
    "sourceSetName": "commonMain"
  },
  {
    "moduleId": "module androidx.collection:collection:1.5.0",
    "file": "/usr/local/google/home/aurimas/Code/androidx-main/out/androidx/navigation/navigation-common/build/kotlinTransformedMetadataLibraries/commonMain/androidx.collection-collection-1.5.0-commonMain-j7f1lg.klib",
    "sourceSetName": "commonMain"
  },
  {
    "moduleId": "module org.jetbrains.kotlinx:kotlinx-serialization-core:1.7.3",
    "file": "/usr/local/google/home/aurimas/Code/androidx-main/out/androidx/navigation/navigation-common/build/kotlinTransformedMetadataLibraries/commonMain/org.jetbrains.kotlinx-kotlinx-serialization-core-1.7.3-commonMain-s2qClw.klib",
    "sourceSetName": "commonMain"
  }
]

so we don't even have it

[aurimas]

i've spent time trying to make a standalone repro but I haven't had a ton of luck yet

[aurimas]

ok, it looks like you can repro with GitHub copy of androidx

[aurimas]

https://youtrack.jetbrains.com/issue/KT-84533/KGP-regression-for-compileCommonMainKotlinMetadata

[aurimas]

let me know if I can help debug this further

[anton.lakotka]

Hi, so with dependencies it is quite tricky.
Indeed problem is caused by "stricter" dependency resolution.

In your case androidx.savedstate:savedstate:1.4.0 is upgraded to project dependency for androidMain source set through project dependency on :savedstate:savedstate-ktx

./gradlew :navigation:navigation-common:dependencies -q --configuration androidCompileClasspath 

------------------------------------------------------------
Project ':navigation:navigation-common'
------------------------------------------------------------

androidCompileClasspath - Compile classpath for 'android/main'.
....
+--- project :savedstate:savedstate-ktx
|    +--- project :savedstate:savedstate
+--- androidx.savedstate:savedstate:1.4.0 -> project :savedstate:savedstate (*)
.....

this is not the case of other platforms, and commonMain dependencies.

So when we try to resolve dependencies for commonMain KGP matches resolved dependencies from commonMain and its underlying targets.
And it failed to find resolved savedstate as external component dependency for android main, and algorithm silently "skips" this dependency. (that is not good 😞 )

And ideally we need to detect this case and allow matching "savedstate" as external with "savedstate" as project dependency.

But in first place I'd suggest to adjust dependencies on your side, as, in general, it is not good when one dependency is resolved to different versions on different platforms.
We're doing consistency resolution among "metadata configurations" but with platform ones it was problematic, but still desirable.
https://kotlinlang.org/docs/multiplatform/multiplatform-advanced-project-structure.html#aligning-versions-of-common-dependencies-across-source-sets
https://youtrack.jetbrains.com/issue/KT-66375/KMP-Make-consistent-dependency-resolution-for-Common-and-Platform-source-sets

Do you have more problems like this? or was it a unique one?

[aurimas]

So far the only one. How do I detect this problem in other projects?

[aurimas]

in general, it is not good when one dependency is resolved to different versions on different platforms.

We do this all over the place

[aurimas]

Especially because WASM/js force newer things

[aurimas]

e.g. kotlin-stdlib has to be the latest for those platforms, but we don't want to force that our unsuspecting JVM/android users

[aurimas]

same is true for other libraries

[aurimas]

I think the question still stands on why dependency disappears though

[anton.lakotka]

> We do this all over the place
that is why we can't solve KT-66375 right away, perhaps, it will be solved in a different manner many years ahead 🙂

> e.g. kotlin-stdlib has to be the latest for those platforms, but we don't want to force that our unsuspecting JVM/android users
Yes, this is one of the reasons why it is necessary.

> I think the question still stands on why dependency disappears though
So I have answer why.

androidx.savedstate:savedstate:1.4.0 is resolved to different components for commonMain and androidMain.
for commonMain it is androidx.savedstate:savedstate:1.4.0
But for androidMain it is project :savedstate:savedstate

Before 2.3.20, metadata artifact transformation was lenient to such mismatches and it was still building expected classpath.
Since 2.3.20 I 'fixed' this leniency and now the algorithm has to find correct matches. between common and platform dependency resolutions.

However the matcher is buggy, as it can't match ``androidx.savedstatesavedstate1.4.0`` with ``project savedstatesavedstate`` as gradle did.

I think for you to workaround this problem it should be enough to change commonMain dependency from androidx.savedstate:savedstate:1.4.0 to project(":savedstate:savedstate")

This way commonMain (and other platforms) will resolve this dependency to the same component as androidMain. (in YT issue you wrote that it worked).

So question is, do we need to fix it in 2.3.20? or it can wait till 2.4 (or if needed 2.3.21)?

[aurimas]

I am working around by changing it to project dependency, but I hit more build failures, I need to spend time to see if they are our own fault or if they are new regressions

[aurimas]

I think i'm ok with dependency swap for 2.3.20, but you need to make it very clear in your release notes about this known shortcoming

[aurimas]

I think most people would struggle to figure this out

[anton.lakotka]

but you need to make it very clear in your release notes about this known shortcoming

I think most people would struggle to figure this out

yes, I agree.

[anton.lakotka]

Let me know if you can confirm that more issues like this appear in your build.
I'll ask release team for 2.3.20 fix.

[aurimas]

I'll get back to you hopefully end of my day. Our builds are very large so moving very slowly

[anton.lakotka]

I understand