https://kotlinlang.org logo
Join Slack
Powered by
Hey guys! I have a mono repo with lots of gradle m...
# library-development
r
Hey guys! I have a mono repo with lots of gradle modules. I now want to publish some of them as external libraries (really just one, but that transitively will require others). The issue is that I want to be on the latest Kotlin version internally, but allow users of my library to use letโ€™s say Kotlin 1.8. (What in your view is a reasonable min requirement at this point btw?) Afaik I can only have one Kotlin gradle plugin in the classpath. So does this require a gradle composite build? Or is it enough to set api and language compatibility and make sure to depend on older versions of f.ex ktor, serialisation etc? (Even that seems annoying as I will have two versions of the same libs in libs.versions.toml right? ๐Ÿค”)
m
Use latest KGP with 
lamguageVersion
and 
coreLibrariesVersion
.
As you found out, you also need yo make sure you api dependencies are all compatible with Kotlin 1.8. they can have 1.9 metadata thanks to the n+1 forward compatibility of kotlinc
r
Only api dependencies? Yesterday I was doing some tests and I got into an issue with kotlin std lib. Unless I missed something and leaked it somehow?
m
Kotlin stdlib is an api dependency by default
It's added automatically by KGP
r
Oh.. and it doesnโ€™t match my language and api version compatibility settings? ๐Ÿค”
m
It matches 
coreLibrariesVersion
r
Hmm interesting I did not set that one
m
Not 
apiVersion
there are youtracks about this
You set it in the kotlin {} block directly
๐Ÿ‘ 1
r
So if I user ktor internally as implementation detail, then I can use latest version?
m
Double check but I would say so
๐Ÿฆธ 1
r
Ok thank you ๐Ÿ™ I will try
๐Ÿ‘ 1
Just curious.. how does it work? Ktor depends on stdlib 2.1.x and my user on 1.8.x. The final one bundled in my userโ€™s build would be 2.1.x. Wouldnโ€™t that at the very least possibly cause issues at runtime related to breaking changes? Or are we relying on the strict compatibility guarantees of such core library? (Meaning, they probably avoid breaking changes like the plague ๐Ÿ˜œ )
m
kotlin-stdlib is super stable, substituting a higher version at runtime is no problem at all
Also 2.x is completely backward compatible with 1.x
r
Ok, nice! Just one last question: If my main module that gets imported by the user uses module B which does have 
api(libs.ktor)
is it an issue in this case?
m
The day they break something, we'll have a lot of issue. But that day hasn't come and will probably never come
r
makes sense ๐Ÿ‘
m
If your module B is an implementation dependency, you should be fine
โœ… 1
What matters is the compile classpath. You should be able to see this with 
./gradlew dependencies
. IIIRC, the configuration to check is the 
apiElements
one
๐Ÿ†— 1
thank you color 1
r
thank you so much Martin! You're the real hero ๐Ÿ˜„
m
This is my favorite topic ^^
K 1
r
At first try with maven local, didnโ€™t work. Iโ€™m getting still on the 1.8. test project:
Copy code
e: file:///Users/rafael.costa/personal/ElectricEelSdkTests/app/src/main/java/com/ramcosta/electriceelsdktests/MainActivity.kt:14:15 Class 'kotlin.Unit' was compiled with an incompatible version of Kotlin. The binary version of its metadata is 2.1.0, expected version is 1.8.0.
The class is loaded from /Users/rafael.costa/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib/2.1.20/aa8ca79cd50578314f6d1180c47cbe14c0fee567/kotlin-stdlib-2.1.20.jar!/kotlin/Unit.class
I see lots of references to stdlib 2.1.20 when I run 
./gradlew :main-module:dependencies
but it's kinda hard to interpret this output ๐Ÿ˜… If I search for "apiElements" they all seem "empty". running now with 
--scan
, see if I can spot anything
m
Alright, sorry, looks like 
compileClasspath
is the one
๐Ÿ™Œ 1
You should see there, which dependency is pulling 
kotlin-stdlib:2.1.0
r
It shows there as top level. Weirdly next to a jdk one 1.8.0 (which is the version I forced with 
coreLibrariesVersion
This is a KMP module that uses compose as implementation dependency. Not sure if that matters. I wonder if any other plugin sets stdlib as api like KGP or something
I set this up on my convention pubishing plugin like so:
Copy code
extensions.configure<KotlinMultiplatformExtension> {
                    coreLibrariesVersion = "1.8.0"
                    compilerOptions {
                        apiVersion.set(KotlinVersion.KOTLIN_1_8)
                        languageVersion.set(KotlinVersion.KOTLIN_1_8)
                    }
                }
m
That looks good ๐Ÿ‘
Looks like something is also adding 
kotlin-stdlib
to your dependencies...
Maybe try the Gradle property:
Copy code
kotlin.stdlib.default.dependency=false
That will rule out KGP as the thing that adds kotlin-stdlib
If that's something else, not sure how to find out besides disabling Gradle plugins progressively and/or putting breakpoints in your build logic
๐Ÿ‘ 1
thank you color 1
r
In the meantime I found I was actually manually adding the stdlib ๐Ÿคฆโ€โ™‚๏ธ That said, as an implementation. Removing it didn't help either. I wonder what I am misunderstanding, since I see things like ktor in the 
compileClasspath
which still depend on stdlib. My understanding was that only things that are part of the module public API would be there right? ๐Ÿค” Also, I've tried forcing the stdlib dependency with something like:
Copy code
configurations.all {
    resolutionStrategy {
        dependencySubstitution {
            substitute(module("org.jetbrains.kotlin:kotlin-stdlib"))
                .using(module("org.jetbrains.kotlin:kotlin-stdlib:1.8.0"))
                .because("whatever")
        }
    }
}
but when compiling with that I get an exception somthing like:
Copy code
java.lang.AssertionError: Type should be an array, but was KClass<out Annotation>: [NormalClass(value=kotlinx/coroutines/InternalForInheritanceCoroutinesApi)]
It looked promising since on 
compileClasspath
I don't see references to 2.1.20 stdlib.
m
Mmm right. Sorry I might have misled you. compileClasspath is indeed the classpath for your module, not what is exported by your module
So apiElements is probably the one but maybe it's not in "dependencies"....
What does 
./gradlew outgoingVariants
say?
Sorry about this, I need more coffee
r
ahah no worries at all, I'm glad you're even here ๐Ÿ™‡ ๐Ÿ™
๐Ÿ’™ 1
a lot of stuff, but nothing about stdlib or version 2.1
not sure what i'd be looking for ๐Ÿ˜ฌ I really need to go for now, but thank you once again. Let me know if you think of something else, I'll try when I'm back. Thank you once again!
m
I haven't found a way to dump that information without actually modifying the build files. But you can do this:`
Copy code
configurations.resolvable("exportedCompileClasspath") {
  extendsFrom(configurations.apiElements.get())
}
And then
Copy code
./gradlew :modul:dependencies --configuration exportedCompileClasspath
If you're KMP, you might have to tweak a bit the 
apiElements
naming, probably use 
jvmApiElements
Another option is to call 
./gradlew publishToMavenLocal
and then look inside the 
.module
file if you have publishing configured
I asked the question in the Gradle slack if you're there: https://gradle-community.slack.com/archives/CAHSN3LDN/p1748692460738109
thank you color 1
โœ… 1
r
Yes I am ๐Ÿ‘Œ And I have publishing setup at least for maven local, thatโ€™s how Iโ€™m testing it. Iโ€™ll take a look when Iโ€™m home again.
๐Ÿ‘ 1
m
I added a tool to 
compatPatrouille
to check for that: โ€ข https://github.com/GradleUp/compat-patrouille/pull/2
r
I actually went to my project consuming this module and did a 
gradle :app:dependencies
(after not seeing any mention to stdlib 2.1 in .module files that I'm expecting to be used directly). Is it possible that runtimeClasspath actually does matter for what stdlib gets picked up in the end? Meaning that even if all my modules that use 2.1 are not part of my API 2.1 could still be required? this image seems to suggest that (I just expanded the 
debugCompileClasspath
)
I assume the 
->
arrows mean 
<version specified> -> <version after resolution>
right?
In fact, the more I think about the more confused I get... We said there are no breaking changes, but thats only true in a "background compatibility" way. They do add APIs to stdlib, so if I manage to compile an app using an older stdlib but still using a newer transitive dependency that uses one of those new APIs, it would crash at runtime, correct? ๐Ÿค”
m
The runtime substitution is most likely fine. I'm guessing the main problem you have is kotlinc trying to decode unsupported metadata
Ktor can use 2.1 symbols at runtime but your module apiElements only expose 1.9
Dependents of your module can compile with 1.9
r
so you mean, at the end, at runtime 2.1 would be the one present, right?
it has to
m
Yes
r
alright ๐Ÿค”
m
That's the difference between compileClasspath and runtimeClasspath
๐Ÿ‘ 1
r
on the consuming app, running 
gradlew :app:dependencies --configuration debugCompileClasspath
Gives me a bunch of this:
Copy code
+--- org.jetbrains.kotlin:kotlin-stdlib:1.8.22 -> 2.1.20 (*)

OR

\--- org.jetbrains.kotlin:kotlin-stdlib-common:2.1.20 (c)

OR a couple of (without a left side of the arrow version)

+--- org.jetbrains.kotlin:kotlin-stdlib -> 2.1.20 (*)
and a these two in the end as top level:
Copy code
+--- org.jetbrains.kotlin:kotlin-stdlib:{strictly 2.1.20} -> 2.1.20 (c)
+--- org.jetbrains.kotlin:kotlin-stdlib-common:{strictly 2.1.20} -> 2.1.20 (c)
m
Interesting ๐Ÿค”
r
As soon as I remove my module run that same command, all those go away, so it's definitely something on it. Maybe it's me, but this tree is not easy to parse ๐Ÿ˜…
m
It has 2 modules: โ€ข 
lib
uses KGP 2.1 and has ktor as an 
implementation
dependency โ€ข 
app
uses KGP 1.8 and has lib as a dependency
If I move ktor to an 
api
dependency, then the build fails with
Copy code
e: file:///Users/martinbonnin/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-serialization-core-jvm/1.8.1/510cb839cce9a3e708052d480a6fbf4a7274dfcd/kotlinx-serialization-core-jvm-1.8.1.jar!/META-INF/kotlinx-serialization-core.kotlin_moduleModule was compiled with an incompatible version of Kotlin. The binary version of its metadata is 2.1.0, expected version is 1.8.0.
But as long as it stays 
implementation
, it is fine
r
damn...
it's gotta be something else here
then
m
debugCompileClasspath
So this is an Android app, maybe something with AGP?
Maybe try
Copy code
afterEvaluate {
  afterEvaluate {
     println(kotlin.coreLibrariesVersion)
  }
}
Make sure no one touches that?
r
I'm now looking at 
androidx.core:core-ktx
seems to have been forced to 
1.13.1
which uses stdlib 2.1
so yeah maybe related to agp auto including some api dependency ๐Ÿค”
m
Is 
androidx.core:core-ktx
an 
api
dependency?
r
not explicitly by me
Maybe try
```afterEvaluate {
afterEvaluate {
println(kotlin.coreLibrariesVersion)
}
}```
2.1.20 ๐Ÿ˜ฎ
nvm that was on another branch ๐Ÿ˜› gotta take a break ahah
โ˜• 1
So, not that this is a final solution, but it could be a clue. I managed to make the app work by both forcing coroutines version and excluding std lib from my module as I include it:
Copy code
resolutionStrategy {
    force 'org.jetbrains.kotlinx:kotlinx-coroutines-core-jvm:1.9.0'
    force 'org.jetbrains.kotlinx:kotlinx-coroutines-android:1.9.0'
}
AND
Copy code
implementation('com.teya.pos:unified-sdk-core:0.0.1-internal37') {
    exclude group: 'org.jetbrains.kotlin', module: 'kotlin-stdlib'
}
(yeah using groovy in this "old versions" project ๐Ÿ˜›)
๐Ÿ‘€ 1
and if I do only one of these two, it doesn't work.
Is it possible that I'm "leaking" coroutines somehow in my APIs? Got that idea from our overlords AIs ๐Ÿ˜„ But I'm even using 
explicitApi()
mode... will dive into it.. even though it seems like it wouldn't be enough to fix it? ๐Ÿคท
m
Looks like something in your lib is still leaking stdlib but it's hard to know exactly what. Any chance this is open source? I could look into it
r
It lives in a closed repo, but we actually will include sources.jar for the dev experience, so it's not closed either I guess ๐Ÿ˜›
is there a tool I can run that lists all my public APIs? Like f.ex the binary compatibility thing that I've heard of but never used? ๐Ÿ˜„
hmm If I have a public class annotated with @Serializable, am I leaking serialization stuff? ๐Ÿ˜›
oh yes.. I can see the Companion object that gets generated on my "old versions app"... ๐Ÿ‘€
m
This plugin does wonders but actually in the JVM, what you expose in your public api symbols and what you expose in your gradle dependencies are not enforced
This is the problem that Tony's plugin is solving
But you may very well expose a 2.1 symbol in your public api while still having 1.9 in your apiElements
What I'm trying to say is, if your problem is a metadata error, Binary Compatibility Validator and/or Tony's plugin won't really solve it. But that can help getting a better understanding of what's going on.
๐Ÿ‘ 1
r
Will definitely take a look. Thank you
Letโ€™s hope this was it, but was a good catch regardless and likely at least one reason for what I was seeing
What I newb thing to do btw shame on me ๐Ÿคฆโ€โ™‚๏ธ
I got nothing left on me ahah, I give up (โ•ฏยฐโ–กยฐ๏ผ‰โ•ฏ๏ธต โ”ปโ”โ”ป For some reason that plugin won't show me hints for my kmp modules, not sure if it works with kotlin multiplatform plugin. Also added the kotlin binary compatibility validator and went through all my public API. All I found which I am not sure about is โ€ข lambda related stuff like 
Lkotlin/jvm/functions/Function1
(not sure if it could cause stdlib? but would be weird) โ€ข Some Composables which I marked as internal, but still seem like they leave some residue - also very weird, but not sure if could be the cause ๐Ÿค” , Like:
Copy code
public final class com/teya/unifiedepossdk/internal/poslink/ui/stores/ComposableSingletons$NoStoresScreenKt {
	public static final field INSTANCE Lcom/teya/unifiedepossdk/internal/poslink/ui/stores/ComposableSingletons$NoStoresScreenKt;
	public fun <init> ()V
	public final fun getLambda$1016637691$core ()Lkotlin/jvm/functions/Function2;
}
All this file has f.ex is a single internal @Composable function ๐Ÿคท
Still have no idea why it seems like as soon as I add my module in another project, it tries to strictly update coroutines and stdlib separately (as in, coroutines need stdlib 2.1 but 2.1.20 is there for whatever reason).
@mbonnin I am now reducing the number of dependencies to try to reduce the possible reasons, and I have a theory which seems to align with what I am seeing. Would like your input on it ๐Ÿ˜› โ€ข I have coroutines in my 
lib
module as 
implementation
version 1.10.2. โ€ข If I check 
dependencies --configuration debugCompileClasspath
from app project importing 
lib
it is almost empty within the 
lib
tree - no mention on coroutines. โ€ข If I do 
gradle -q app:dependencyInsight --dependency org.jetbrains.kotlinx:kotlinx-coroutines-core --configuration debugCompileClasspath
I see this reason:
Selection reasons:
- By constraint: debugRuntimeClasspath uses version 1.10.2
- By ancestor
My theory: โ€ข If the user of my lib also depends on the same library (in this case coroutines), then that library is put on 
compileClasspath
โ€ข Then gradle tries to use a consistent version between runtime and compile classpaths, so it ends up still choosing the newest one (1.10.2 in this case). โ€ข If the user would not depend on coroutines, then like you suggested at compile time the version I use does not matter, coroutines would not even be in the conversation.
(omg, sorry for tagging this late, I didn't even notice the time ๐Ÿ˜› Hope I'm not bothering! I'll get some sleep I guess ๐Ÿ˜… ) Thanks in advance! ๐Ÿ™
m
Good catch, indeed that sounds like the issue
I haven't been able to reproduce in a JVM only project though so that would point at AGP ๐Ÿค”
Now why would they do this is an interesting question...
Copy code
org.jetbrains.kotlinx:kotlinx-coroutines-bom:{strictly 1.10.2} -> 1.10.2
Who's adding the coroutines BOM ๐Ÿค” ?
r
Apparently Android core ktx
Most Android apps will have that (it even comes on the template project in the IDE)
So theyโ€™ll all have coroutines in the compileClasspath
So basically, if I want to truly support older Kotlin, I guess I have to depend on older versions even at runtime
Core ktx probably leaks coroutines because it exposes suspend functions or similar, but I guess thatโ€™s expected and even wanted.
m
Copy code
// make compileClasspath match runtimeClasspath
        compileClasspath.shouldResolveConsistentlyWith(runtimeClasspath);
that's .... surprising
Looks like you can set 
android.experimental.dependency.excludeLibraryComponentsFromConstraints
to disable this
r
Interesting.. so youโ€™re saying that on a jvm non Android based project, compile and runtime classpaths can use different versions?
m
Yep
This is really how it's supposed to be
If not, might as well only one classpath...
r
Ok.. I was actually about to ask why they had to match to begin with
m
Why is AGP overriding this, I have no clue
There has to be a good reason, still digging the git history
r
getting 400 on that url โ˜๏ธ
m
Maybe try to sign in?
r
๐Ÿคฆโ€โ™‚๏ธ I was using another browser account my bad
seeing it now ๐Ÿ‘
๐Ÿ‘ 1
with this gradle property, I'm getting:
Copy code
* What went wrong:
Execution failed for task ':app:checkDebugClasspath'.
> Cannot fingerprint input property 'compileVersionMap': value '{Info(group=org.jetbrains.kotlin, module=kotlin-stdlib-jdk8)=1.8.0, ........}' cannot be serialized.
I added it only in the consuming project. In your reproducer it is applied also to the lib I'm assuming? ๐Ÿค” Not sure if that matters.
Also tried using
Copy code
android.enableCompileRuntimeClasspathAlignment=false
as seen on that google issue, but it apparently has no effect whatsoever here..
m
checkDebugClasspath
Sounds like you're using an older AGP?
Maybe <8 ?
r
I am yes.
Trying to simulate what my clients would have
m
This was changed at some point. Don't really have the time to do the archeology but you'll have to see how to adapt this to older "AGP"
๐Ÿ‘ 1
r
ofc no worries ๐Ÿ™‚
m
Maybe "just" disable the 
checkDebugClasspath
task?
r
thank you so much ๐Ÿ™
m
Sure thing!
r
at least we found the root cause I would say!
m
This is starting to feel like a lot of trouble for backward compatibility though...
Even though backward compatibility is always nice of course
r
I'm guessing any workaround we find on the consumer side is not really a thing we want to go with. So, likely, we'll have to just use older versions at runtime too for the published modules ๐Ÿคท
m
Yea, probably what I would err into as well if you really care about AGP 7 users
Or make a split artifact
But that's some work...
r
Yeah.. ideally we would just document our version requirements, but not sure if we can afford to just leave some possible clients off the table ๐Ÿ˜ž
m
Yea, that's a tough tradeoff to make...
r
coz this came from a real 3rd party partner trying to use the SDK. They're even on older versions btw, but I don't think we can reasonably go down to kotlin 1.4 ahah So I'm just trying to find what's the lowest we can support.
and hopefully our partners can reach us in the middle. I'm thinking kotlin 1.8 should be reasonable..
m
The struggle is real...
r
yeah ๐Ÿ˜ž Btw I had been totally focused on this SDK for weeks and its promise was soooo good.
Thanks to KMP and Compose we were able to do it so quickly and provide a lot of value on all platforms (Android, iOS, Desktop)
but then a real consumer comes in to try an early EAP version... and.... BOOOM
my world crumbled ahaha xD
Still crying inside ๐Ÿ˜ข
m
kodee sad
It'll get there...
At some point
r
The biggest limitation is Compose ofc. Because we need the latest improvements and features. So the solution will have to be: โ€ข Split the SDK in Core and UI โ€ข Core can support down to kotlin 1.8 โ€ข UI will be usable only from kotlin 2.0 onward โ€ข If partners can't update kotlin, they'll have to implement the UI part on their side ๐Ÿคท
๐Ÿ‘ 1
For what is worth, KMP and CMP will still provide a LOT of value for iOS and Windows, where I don't think we'll have these issues. So still โค๏ธ K
โค๏ธ 1
m
I think that's fair
r
Thanks once again Martin, you've been amazing. Cheers ๐Ÿค
m
Thanks for the interesting use case and deep dive!
โค๏ธ 1
9 Views
Open in Slack
Previous
Powered by