https://kotlinlang.org logo
#gradle
Title
# gradle
c

CLOVIS

09/24/2023, 6:29 PM
Is it normal that
publishAllPublicationsToCentralRepository
for a multiplatform project creates multiple staging repositories on Nexus? The
klib
of one of the modules is sent to its own staging repository. Even the sha1 etc files are sent to the other, so closing fails with "missing checksum validation".
m

mbonnin

09/24/2023, 6:33 PM
Definitely not normal but I've seen that happen sometimes. I call this "split repos". You can use https://github.com/gradle-nexus/publish-plugin to force an id or you can do it manually too
c

CLOVIS

09/24/2023, 6:34 PM
…I see that on literally all uploads I've ever made. It's the first time the signatures are not in the same repo as their files though 😕
m

mbonnin

09/24/2023, 6:35 PM
Since there is no "transaction" concept, I'm guessing the sonatype servers use heuristics based on the host machine, time and IP to determine what belongs to the same staging repo. If you publish from different machine, this is more likely to happen but I've seen this with single machines too 🤷
🤔 1
e

ephemient

09/24/2023, 6:38 PM
for publishing multiplatform libs to sonatype, the way I've seen is to publish to a temporary local repo first and then to do a bulk upload of that to sonatype
which lets you build on different machines for different platforms too
c

CLOVIS

09/24/2023, 6:43 PM
publish to a temporary local repo first and then to do a bulk upload of that to sonatype
Do you have an example of how to do that?
If you publish from different machine, this is more likely to happen but I've seen this with single machines too
I wonder if it happens because I have the configuration cache enabled, and thus is does multiple uploads in parallel 🤔
m

mbonnin

09/24/2023, 6:43 PM
maybe
I'd recommend you allocate a staging repo yourself before uploading though. Even if the local repo thing works, it's still fragile and relies on Sonatype heuristics
c

Casey Brooks

09/24/2023, 6:48 PM
It’s an issue with Nexus, not with Gradle publication, basically a race condition from what I understand. Gradle publishes in parallel, which means the artifacts may randomly end up creating a new staging repository, or it may be added to an existing one. The way I go about it is to explicitly call the Sonatype APIs to create the staging repo before publication and cache its generated ID, and then use that ID in the configured repository URL. You can see how I set it up here, here, and here. The publication process then becomes
Copy code
./gradlew openSonatypeStagingRepository
./gradlew publish
./gradlew closeSonatypeStagingRepository

then manually release the builds from Nexus
c

CLOVIS

09/24/2023, 6:49 PM
Do you have an example of usage of gradle-nexus in a multi-project build? The documentation states
The plugin must be applied to the root project
but then all the examples use the
publishing
DSL to configure the POM etc, which definitely cannot be in the root project since it doesn't have any code/artifacts…
c

Casey Brooks

09/24/2023, 6:50 PM
As a bonus, this method is also robust enough that it works if you’re publishing builds from multiple CI agents for different OSs. For example, first stage open the repo and saves the result as a build artifact. The next stages pull down that ID, and publish their builds. Finally, a 3rd stage closes the repo
👀 2
m

mbonnin

09/24/2023, 6:52 PM
Do you have an example of usage of gradle-nexus in a multi-project build?
I don't sadly, I'm trying to handle the publishing with as little dependencies as possible. Every publishing plugin out there does some things slightly differently. But I wrote a small Kotlin client that opens/closes a staging repo so it's a few lines to do so https://github.com/martinbonnin/vespene
I then store it as a extra property of the root project so it can be reused later on
c

CLOVIS

09/24/2023, 6:53 PM
Everyone has a different way to do things, apparently 😅 no wonder it's so hard to learn how to do it
m

mbonnin

09/24/2023, 6:53 PM
Yea, it's a mess
I'll add one more option 😄 . With @romainbsl we've been writing https://github.com/nexus-actions/nexus-actions-demo. It's handy if you have your builds scattered in different CI VMs
🙌 1
c

Casey Brooks

09/24/2023, 6:54 PM
Yup. the process is a horrible mess, and the plugins that do Sonatype publishing either don’t do it correctly (because they’re made for Java projects, not Kotlin), or are so over-architected and difficult to configure that it’s easier to just write all that logic yourself.
💯 1
c

CLOVIS

09/24/2023, 6:54 PM
Ah, I'm on the GitLab team though 😇
m

mbonnin

09/24/2023, 6:55 PM
Sounds like a great opportunity to add one more options to the mix 😁
🤣 1
😅 1
c

CLOVIS

09/24/2023, 6:56 PM
@Casey Brooks are your convention plugins published anywhere, and if so, do you think they are usable in other projects as they are? Since convention plugins tend to quite opinionated. Otherwise, I'll just end up copying them in my own.
c

Casey Brooks

09/24/2023, 7:00 PM
There’s a few things hardcoded in it, but you’re welcome to fork this repo and make the necessary tweaks to use it yourself. I develop it just to keep Gradle in sync between my libraries and personal projects, but it’s basically fully set up for publishing KMP Compose projects as simply as possible. Each configuration plugin is kept as close to what you’d find in a regular
build.gradle.kts
file as possible, so doesn’t have too many surprises in it. It’s also not documented, but you can see how it’s used from my other libraries, like Ballast which is usually the one kept the most up-to-date with the convention plugins
c

CLOVIS

09/24/2023, 7:01 PM
I see. I'm doing the same over here: https://gitlab.com/opensavvy/playgrounds/gradle
Uh, interesting,
plugins { id("…") apply false }
is not transitive: if you add it in a convention plugin, and apply the convention plugin, it applies that plugin as well.
m

mbonnin

09/24/2023, 7:20 PM
This sounds.... unexpected...
c

CLOVIS

09/24/2023, 7:21 PM
I tried to put the
kotlin("multiplatform") apply false
in a convention plugin for the root project, and well, that doesn't work
m

mbonnin

09/24/2023, 7:22 PM
There's very little value in doing ``plugins { id("…") apply false }`` in a convention plugin. Usually you would do
apply false
to pull the plugin jar in the buildscript classpath
But in a convention plugin, you control the buildscript classpath through
build-logic/build.gradle.kts#dependencies.implementation()
j

Jeff Lockhart

09/24/2023, 7:40 PM
👀 Following this conversation, as I'm currently setting up Maven Central publication for my KMP library. Thanks for the tips.
c

CLOVIS

09/24/2023, 8:32 PM
🤔 I setup the Gradle Nexus Publish plugin, but it still creates a staging repository for each execution.
My configCI job that creates the repository (does create one) • CI job that uploads the files (creates a new repository) • CI job that's supposed to close the repository (fails because it doesn't find it)
The two repositories that were created do have the same description though…
m

mbonnin

09/24/2023, 8:38 PM
Can the Gradle Nexus Publish can reuse an already existing repoId? Looks like it creates one by default?
r

romainbsl

09/25/2023, 7:13 AM
I think your child CI jobs do not use the previously created repositoryId. You need use a specific URL to upload your artifacts, something like
<https://s01.oss.sonatype.org/service/local/staging/deployByRepositoryId/${repositoryId}/>
but I agree it doesn't look like it does
m

mbonnin

09/25/2023, 8:29 AM
It says
different Gradle invocations
, not different VMs. My hunch is that the
initializeStagingRepo
task stores its state (the
repoId
) on the local filesystem
you might want to publish from CI and close and release from your local machine
Ah, maybe not after all 🤔
🤔 1
./gradlew findSonatypeStagingRepository
Looks like it stores state on Sonatype (and therefore allows only one repo at a time?)
So you'd have to call ``findSonatypeStagingRepository`` before each CI job that's not the first one to sync the remote state to the local state
Feels fragile because if you do 2 publish at the same time, you're doomed
the repository can be found later using the same description.
Right so it's a bit better but still feels a bit backward
c

CLOVIS

09/25/2023, 8:38 AM
Looks like it stores state on Sonatype (and therefore allows only one repo at a time?)
What I've seen is that it adds the full name of the artifact, including version, as the description of the staging repo, and I assume it's supposed to use those to find it again later? But in my case, it doesn't seem like it does
Feels fragile because if you do 2 publish at the same time, you're doomed
I guess it's unlikely to do two publishes for the same project and same version at the same time, so it should be safe
m

mbonnin

09/25/2023, 8:50 AM
Indeed sorry, I saw that too late. It's safe enough I guess
Until someone deletes a tag and repushes it 🤷
😅 2
c

CLOVIS

09/25/2023, 8:51 AM
That shouldn't happen in my case
Anyway, I could just add the unique CI pipeline ID to the description, if it did as it says in the README
m

mbonnin

09/25/2023, 8:53 AM
Yea, it's probably "good enough" but still not a fan of the approach. Like if you add Gradle build cache, is
findStagingRepository
going to re-use a previous entry, what are the inputs of the tasks
Maybe if it has no input it's going to be re-executed always
So yea it's mostly working but feels a bit weird
c

CLOVIS

09/25/2023, 9:06 AM
Oh… yeah, that's a very important detail, I do use the remote build cache 😅
m

mbonnin

09/25/2023, 9:15 AM
I think it's okay, would be weird to cache something that doesn't have any inputs. I'm guessing the task is declared as non-cacheable itself. Just feels a bit weird to have build state on Sonatype repo
c

CLOVIS

09/25/2023, 9:05 PM
Latest findings: the Nexus plugin expects to call
:findSonatypeStagingRepository
before executing the
:closeSonatypeStagingRepository
, which will make it find it correctly. However, it doesn't seem to work for the
:publishToSonatype
task, which still creates a new repository. I created an issue → https://github.com/gradle-nexus/publish-plugin/issues/275
j

Jeff Lockhart

09/25/2023, 10:58 PM
I've seen vanniktech's publishing plugin used in a lot of projects. Have you tried it? It lists as an advantage:
It also avoids issues like having multiple staging repositories on Sonatype OSS
a

andylamax

09/26/2023, 1:49 AM
I switched to vanniktech's solution after encountering this multiple repository on sonatype. I have to admit, I never encountered similar problem ever again
🙌🏼 1
c

CLOVIS

09/27/2023, 8:32 PM
Reporting back! Gradle Nexus does work, here's my final setup: https://gitlab.com/opensavvy/playgrounds/gradle/-/merge_requests/23 Glad all of this is finally behind me, and I can actually get to code 😐
🚀 2
nod 1
j

Jeff Lockhart

09/27/2023, 11:49 PM
I've had good success using vanniktech's plugin as well. It works well out of the box, with minimal configuration. I haven't set up CI yet and only published a SNAPSHOT, but looks like my config is working. I'm assuming it's normal for SNAPSHOT releases to not show up in the staging repositories, since they just go straight to the snapshot repository.
👍 1
👍🏾 1