Is it normal that `publishAllPublicationsToCentralRepository` for a multiplatform project creates mu...

# gradle

[CLOVIS]

Is it normal that publishAllPublicationsToCentralRepository for a multiplatform project creates multiple staging repositories on Nexus?

The klib of one of the modules is sent to its own staging repository. Even the sha1 etc files are sent to the other, so closing fails with "missing checksum validation".

[mbonnin]

Definitely not normal but I've seen that happen sometimes. I call this "split repos".
You can use https://github.com/gradle-nexus/publish-plugin to force an id or you can do it manually too

[CLOVIS]

…I see that on literally all uploads I've ever made. It's the first time the signatures are not in the same repo as their files though 😕

[mbonnin]

Since there is no "transaction" concept, I'm guessing the sonatype servers use heuristics based on the host machine, time and IP to determine what belongs to the same staging repo. If you publish from different machine, this is more likely to happen but I've seen this with single machines too 🤷

[ephemient]

for publishing multiplatform libs to sonatype, the way I've seen is to publish to a temporary local repo first and then to do a bulk upload of that to sonatype

[ephemient]

which lets you build on different machines for different platforms too

[CLOVIS]

publish to a temporary local repo first and then to do a bulk upload of that to sonatype

Do you have an example of how to do that?

[CLOVIS]

If you publish from different machine, this is more likely to happen but I've seen this with single machines too

I wonder if it happens because I have the configuration cache enabled, and thus is does multiple uploads in parallel 🤔

[mbonnin]

maybe

[mbonnin]

I'd recommend you allocate a staging repo yourself before uploading though. Even if the local repo thing works, it's still fragile and relies on Sonatype heuristics

[Casey Brooks]

It’s an issue with Nexus, not with Gradle publication, basically a race condition from what I understand. Gradle publishes in parallel, which means the artifacts may randomly end up creating a new staging repository, or it may be added to an existing one.

The way I go about it is to explicitly call the Sonatype APIs to create the staging repo before publication and cache its generated ID, and then use that ID in the configured repository URL. You can see how I set it up here, here, and here.

The publication process then becomes

./gradlew openSonatypeStagingRepository
./gradlew publish
./gradlew closeSonatypeStagingRepository

then manually release the builds from Nexus

[CLOVIS]

Do you have an example of usage of gradle-nexus in a multi-project build? The documentation states

The plugin must be applied to the root project

but then all the examples use the publishing DSL to configure the POM etc, which definitely cannot be in the root project since it doesn't have any code/artifacts…

[Casey Brooks]

As a bonus, this method is also robust enough that it works if you’re publishing builds from multiple CI agents for different OSs. For example, first stage open the repo and saves the result as a build artifact. The next stages pull down that ID, and publish their builds. Finally, a 3rd stage closes the repo

[mbonnin]

Do you have an example of usage of gradle-nexus in a multi-project build?

I don't sadly, I'm trying to handle the publishing with as little dependencies as possible. Every publishing plugin out there does some things slightly differently.
But I wrote a small Kotlin client that opens/closes a staging repo so it's a few lines to do so https://github.com/martinbonnin/vespene

[mbonnin]

I then store it as a extra property of the root project so it can be reused later on

[CLOVIS]

Everyone has a different way to do things, apparently 😅 no wonder it's so hard to learn how to do it

[mbonnin]

Yea, it's a mess

[mbonnin]

I'll add one more option 😄 . With @romainbsl we've been writing https://github.com/nexus-actions/nexus-actions-demo. It's handy if you have your builds scattered in different CI VMs

[Casey Brooks]

Yup. the process is a horrible mess, and the plugins that do Sonatype publishing either don’t do it correctly (because they’re made for Java projects, not Kotlin), or are so over-architected and difficult to configure that it’s easier to just write all that logic yourself.

[CLOVIS]

Ah, I'm on the GitLab team though 😇

[mbonnin]

Sounds like a great opportunity to add one more options to the mix 😁

[CLOVIS]

@Casey Brooks are your convention plugins published anywhere, and if so, do you think they are usable in other projects as they are? Since convention plugins tend to quite opinionated.

Otherwise, I'll just end up copying them in my own.

[Casey Brooks]

There’s a few things hardcoded in it, but you’re welcome to fork this repo and make the necessary tweaks to use it yourself. I develop it just to keep Gradle in sync between my libraries and personal projects, but it’s basically fully set up for publishing KMP Compose projects as simply as possible. Each configuration plugin is kept as close to what you’d find in a regular build.gradle.kts file as possible, so doesn’t have too many surprises in it.

It’s also not documented, but you can see how it’s used from my other libraries, like Ballast which is usually the one kept the most up-to-date with the convention plugins

[CLOVIS]

I see. I'm doing the same over here: https://gitlab.com/opensavvy/playgrounds/gradle

[CLOVIS]

Uh, interesting, plugins { id("…") apply false } is not transitive: if you add it in a convention plugin, and apply the convention plugin, it applies that plugin as well.

[mbonnin]

This sounds.... unexpected...

[CLOVIS]

I tried to put the kotlin("multiplatform") apply false in a convention plugin for the root project, and well, that doesn't work

[mbonnin]

There's very little value in doing ``plugins { id("…") apply false }`` in a convention plugin. Usually you would do apply false to pull the plugin jar in the buildscript classpath

[mbonnin]

But in a convention plugin, you control the buildscript classpath through build-logic/build.gradle.kts#dependencies.implementation()

[Jeff Lockhart]

👀 Following this conversation, as I'm currently setting up Maven Central publication for my KMP library. Thanks for the tips.

[CLOVIS]

🤔 I setup the Gradle Nexus Publish plugin, but it still creates a staging repository for each execution.

[CLOVIS]

My config • CI job that creates the repository (does create one) • CI job that uploads the files (creates a new repository) • CI job that's supposed to close the repository (fails because it doesn't find it)

[CLOVIS]

The two repositories that were created do have the same description though…

[mbonnin]

Can the Gradle Nexus Publish can reuse an already existing repoId? Looks like it creates one by default?

[romainbsl]

I think your child CI jobs do not use the previously created repositoryId. You need use a specific URL to upload your artifacts, something like <https://s01.oss.sonatype.org/service/local/staging/deployByRepositoryId/${repositoryId}/>

[CLOVIS]

@mbonnin well, the README says it can: https://github.com/gradle-nexus/publish-plugin#publishing-and-closing-in-different-gradle-invocations

[CLOVIS]

but I agree it doesn't look like it does

[mbonnin]

It says different Gradle invocations, not different VMs. My hunch is that the initializeStagingRepo task stores its state (the repoId) on the local filesystem

[mbonnin]

you might want to publish from CI and close and release from your local machine

Ah, maybe not after all 🤔

[mbonnin]

./gradlew findSonatypeStagingRepository

Looks like it stores state on Sonatype (and therefore allows only one repo at a time?)

[mbonnin]

So you'd have to call ``findSonatypeStagingRepository`` before each CI job that's not the first one to sync the remote state to the local state

[mbonnin]

Feels fragile because if you do 2 publish at the same time, you're doomed

[mbonnin]

the repository can be found later using the same description.

Right so it's a bit better but still feels a bit backward

[CLOVIS]

Looks like it stores state on Sonatype (and therefore allows only one repo at a time?)

What I've seen is that it adds the full name of the artifact, including version, as the description of the staging repo, and I assume it's supposed to use those to find it again later? But in my case, it doesn't seem like it does

[CLOVIS]

Feels fragile because if you do 2 publish at the same time, you're doomed

I guess it's unlikely to do two publishes for the same project and same version at the same time, so it should be safe

[mbonnin]

Indeed sorry, I saw that too late. It's safe enough I guess

[mbonnin]

Until someone deletes a tag and repushes it 🤷

[CLOVIS]

That shouldn't happen in my case

[CLOVIS]

Anyway, I could just add the unique CI pipeline ID to the description, if it did as it says in the README

[mbonnin]

Yea, it's probably "good enough" but still not a fan of the approach. Like if you add Gradle build cache, is findStagingRepository going to re-use a previous entry, what are the inputs of the tasks

[mbonnin]

Maybe if it has no input it's going to be re-executed always

[mbonnin]

So yea it's mostly working but feels a bit weird

[CLOVIS]

Oh… yeah, that's a very important detail, I do use the remote build cache 😅

[mbonnin]

I think it's okay, would be weird to cache something that doesn't have any inputs. I'm guessing the task is declared as non-cacheable itself. Just feels a bit weird to have build state on Sonatype repo

[CLOVIS]

Latest findings: the Nexus plugin expects to call :findSonatypeStagingRepository before executing the :closeSonatypeStagingRepository , which will make it find it correctly. However, it doesn't seem to work for the :publishToSonatype task, which still creates a new repository.

I created an issue → https://github.com/gradle-nexus/publish-plugin/issues/275

[Jeff Lockhart]

I've seen vanniktech's publishing plugin used in a lot of projects. Have you tried it? It lists as an advantage:

It also avoids issues like having multiple staging repositories on Sonatype OSS

[andylamax]

I switched to vanniktech's solution after encountering this multiple repository on sonatype. I have to admit, I never encountered similar problem ever again

[CLOVIS]

Reporting back! Gradle Nexus does work, here's my final setup: https://gitlab.com/opensavvy/playgrounds/gradle/-/merge_requests/23

Glad all of this is finally behind me, and I can actually get to code 😐

[Jeff Lockhart]

I've had good success using vanniktech's plugin as well. It works well out of the box, with minimal configuration. I haven't set up CI yet and only published a SNAPSHOT, but looks like my config is working.

I'm assuming it's normal for SNAPSHOT releases to not show up in the staging repositories, since they just go straight to the snapshot repository.