https://kotlinlang.org logo
Join Slack
Powered by
Publishing a Gradle Composite build: What should I...
# library-development
o
Publishing a Gradle Composite build: What should I be using to publish a number of included builds atomically to Maven Central? TestBalloon's newest iteration has 4 publishable included builds. Currently, it uses vanniktech/gradle-maven-publish-plugin with task dependency wiring for publishing tasks. That results in 4 independent publications, and can cause inconsistencies if a subset of them fails. What should I be using instead? • Use JReleaser with 
stagingRepository(...)
? • Use GradleUp/nmcp? • Use GradleUp/librarian? • Anything else?
j
Did you ask that plugin for help? The solution to me seems easy to support: • You run a task to create the staged repo, the ID is returned • You pass that ID into each build as a property • You run a task to close+release the staged repo by ID This is already what the plugin itself is doing for a single build's set of projects.
Although the migration of Central from Nexus their bespoke API may have changed the specifics. I'd be curious what they think about support for it.
o
No, I didn’t ask yet, just scanned issues for hints to no avail. But of course I can!
j
They've been pretty receptive to my feedback with more complex scenarios and on default behavior
o
That’s cool! I was just a bit wary because this scenario doesn’t really look like mainstream, and I didn’t want to push folks down a rabbit hole. But if there’s a chance that it can be easy to accomplish, I should just give it a try.
j
I've worked around this a bit in the past by having the Gradle plugin built twice: once in an included build so the project itself could use it, and then once within the main project solely for publishing
o
Also an option!
j
This was only for a single module though. Yours sounds more complicated. https://github.com/cashapp/redwood/blob/trunk/redwood-gradle-plugin%2Fbuild.gradle#L9-L22
thank you color 1
o
Thanks for the input. I'm adding this to my pool of ideas and also ask the vanniktech folks. One interesting side topic would be if a consolidated publication für included builds could work with snapshot publications. I did not yet look into the details but I wonder about scattered timestamps. Luckily, I have got some time to solve this puzzle as for now I can just live with 4 separate deployments and use a manual step for publishing.
j
I think you could publish all four to a local Maven repo (rootProject/build/maven or something) and then zip that up and upload as a single step.
o
Sure. I already have that local repo for integration tests, but I‘d rather keep the publication process transparent on CI, so I’m just using the MC web interface to switch from „validated“ to „published“.
b
We're still using the good ol' gradle-nexus-publish plugin, but have our internal conventions plugin mangle its task dependencies for KMP. this gives you full flexibility, but has the issue of only supporting the old API. we managed to get it wo work for projects with quite some modules, also publishing artefacts for all targets. NMCP is the natural replacement, because it also does not interfere with creating the actual publications, so it should be as flexible ant tinkerable
m
Semi troll response: Amper doesn't have included builds and won't suffer from this problem (although it's probably not in a state where it can handle a large project like testBalloon)
Could you maybe decrease the number of included builds? They are somewhat required to bootstrap a Gradle plugin but for the others, you can probably to project substitution?
But that still leaves out 2 included builds....
In nmcp, I use an older version of the Gradle plugin in the project to avoid relying on included builds (because not a huge fan of included builds). I have a custom repo that is updated on every push so it's only lagging "1 commit"
Otherwise, yea, publishing everything to a local repo and uploading all at once is probably the path of lowest friction
Nmcp has a low level API to publish arbitrary files. If you manage to get the local repo with all your files, you can use it to create the zip and call the new publishing API
Copy code
nmcp {
  extraFiles(localM2)
}
But you have to manage the local repo yourself, delete stale contents, etc...
o
That wouldn't be a big deal. I'd need to care about signing, as nmcp would not do that for 
extraFiles
, right?
m
Mmm good question, I think it does, let me check
Ah yes I remember now. It doesn't because I didn't find a way to get the signing keys from the 
signing {}
block
So yea you'd have to do it before calling nmcp
👍 1
But if you're using regular 
publishing {}
and 
signing {}
blocks for localM2 it should all work?
o
Regarding the included builds, yes, they have their pain points, but on the publication side, I'd prefer at least one (the Gradle plugin), because that makes TestBalloon better dogfood itself internally. I could drop lots of special 
addTestBalloonPluginFromProject(projects.testBalloonCompilerPlugin)
invocations for examples and other stuff, and just write the build script with the usual plugin block. This also brings better test coverage for these parts. And with one included build, another one (or two) doesn't make a big difference, and helps to avoid back-referencing the top-level build from a subordinate included build.
👍 1
But if you're using regular 
publishing {}
and 
signing {}
blocks for localM2 it should all work?
Yes, I guess nature will find a way. It's just that IIRC, by default, local publications would not use signing (or at least, not insist on it). I need to do some homework.
👍 1
I'm probably holding it wrong: github.com/infix-de/testBalloon/pull/98
👀 1
m
You'll need to add the root project as a dependency for the aggregation:
Copy code
dependencies.add("nmcpAggregation", dependencies.project(":"))
It's a bit silly in this case because it's all the same project but this is the price to pay for isolated/CC/etc... cross project configuration
o
OK, let's try.
m
The second thing is 
populateAggregationStagingRepository.map { it.outputs }
doesn't seem to work
If I run 
./gradlew :outgoingVariants
, I get this:
Copy code
> Cannot convert the provided notation to a File: org.gradle.api.internal.tasks.DefaultTaskOutputs@6f103bf4.
  The following types/formats are supported:
    - A String or CharSequence path, for example 'src/main/java' or '/usr/include'.
    - A String or CharSequence URI, for example 'file:/usr/include'.
    - A File instance.
    - A Path instance.
    - A Directory instance.
    - A RegularFile instance.
    - A URI or URL instance of file.
    - A TextResource instance.
You can do something like this instead:
Copy code
extraFiles(populateAggregationStagingRepository.map { stagingRepository })
o
Yeah, I've written this into the PR comment.
👍 1
Of course I can write it that way, but it doesn't mirror the intention.
m
It would be nice if nmcp failed nicely instead of swallowing the error but the resolution is lenient because this is the way to collect "accross all projects"
Copy code
extraFiles(populateAggregationStagingRepository.map { it.outputs.files })
?
mmm doesn't work either
o
Could you get rid of that "Nmcp: there are no files to publish."? I have
Copy code
extensions.getByType(NmcpExtension::class.java).apply {
    extraFiles(populateAggregationStagingRepository.map { populateAggregationStagingRepository })
}
and
Copy code
dependencies.add("nmcpAggregation", dependencies.project(":"))
both in the root plugin's apply method, and I'm still getting it.
m
Mmm yes I think I got that, I'll open a PR hold on
Re: 
Cannot convert the provided notation to a File: org.gradle.api.internal.tasks.DefaultTaskOutputs@6f103bf4.
it's because the 
extraFiles
are registered as an outgoing artifact so it should be a single file pointing to a directory. I'll fix that
o
🙏 But take your time. I'm not in a rush.
I made a copy-paste error in the above. It should have been 
.map { stagingRepository }
. But fixing that still gave me "there are no files to publish" with 
nmcpZipAggregation
.
m
Yea, I have the same now 🤔 , will get to the bottom of this
👍 1
It was a discrepency
o
Wow! Thanks a lot! Shame on me!
m
No worries, not having good errors doesn't help
o
Yeah, but the combination of stringly-typed and 
Any
-typed is part of the 🐘 ecosystem.
🐘 1
m
Yea...
Opened github.com/GradleUp/nmcp/issues/276 as a follow up. Not 100% sure how typesafe this can be made but worst case I'll clarify the KDocs
8 Views
Open in Slack
PreviousNext
Powered by