This is a joke, right? 3 months to migrate everything off JCenter? How many Java/Kotlin libraries are now unmaintained and won’t get migrated? https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/

Thanks for that. I read it originally but it never clicked that it included jcenter

Yes, what is next since most of the Kotlin ecosystem is on there? And even if all new versions of libraries are released to someplace new it will likely mean a lot of older projects or going back in git history will break after May 1st.

It is is really terrible. Does anybody have any ideas. Publishing to maven central is a pain. I am not sure it is possible to do without controlling top level domain with the correct name

First TravisCI, now Bintray ... what's going on with this world?!

@Jurriaan Mous sadly Space does not provide a single repository even for a single space. Repositories are per-project

TravisCI at least has some decent replacements, and ultimately only impacted your own project. If my dependencies don’t get migrated, I can’t migrate my own projects’ builds

@altavir Ah ok, I am not yet familiar with space but I noticed the Ktor EAP releases where published here: https://maven.pkg.jetbrains.space/public/p/ktor/eap

We are hosting some internal builds in our Space, but as I said, it has the same problem as with github, you have to write a repository for each project and you have to write them in all dependencies

Publishing to Maven Central is not really problematic. The only "hard" part is the initial getting account and being set up. But after that, it just works usually.

It’s problematic in that you need to own the domain for the groupId you’re publishing under, while JCenter did not have that restriction. I’m sure a lot of libraries would need to change their artifact coordinates to be allowed to publish to MavenCentral

I'm concerned about the security implications of all of this. There are some serious risks associated with typo-squatting. And if Central suddenly gets a huge influx of requests for new packages, that's going to seriously overwhelm them.

You don't need to own the domain. There are for example also

com.github.<user>

libraries published on Maven Central

our library currently do not follow domain naming at all. We just use kscience prefix. And I am not sure that I want to change the naming just to satisfy maven-central draconic rules.

Where does it say you need that?

OK, I will check that, thank you

Not necessarily owning the actual TLD, but it does require ownership of the domain or subdomain.

you must choose a groupId for a domain that you own, or for which you are the designated maintainer on behalf of the owner

https://central.sonatype.org/pages/producers.html#individual-projects-open-source-software-repository-hosting-ossrh

The security risks here around typosquatting and namespace hijacking are real. https://blog.autsoft.hu/a-confusing-dependency/

But like @altavir said, I don’t want to have to change to a

io.github.whatever

groupId. It’s long, ugly, and would be a breaking change for the consumers of the dependency if it wasn’t using that already

Hm, I forgot I had to prove I own the domain. Actually it is a bit senseless too, as you can anytime give up the domain or change your GitHub username after you got the namespace assigned. Well, we will see whether Sonatype comes up with some changes in reaction.

Versions submited to Maven Central can never be replaced tho

Versions submited to Maven Central can never be replaced tho

Which usually is a good thing imho 🙂

I think so too! But I just wanted to ensure that if, for example, Maven Central decides to clone JCenter, it won't be a security risk in that sense

The JetBrains could create a Maven Repo for the Community Packages 🤔

It is not cheap to maintain. We just discussed it in telegram and it seems that its much easier to create virtual maven repository that maps requests to small github/space/whatever repositories. It could be based on https://package-search.jetbrains.com/. It won't be quite safe, but it would ensure that all artifacts are resolved with a single repository without a lot of traffic through the server.

I see you point, but I think a oficial Community Repo from JetBrains will help a lot the Kotlin Ecosystem to grow

Indeed, but it could be too expensive

Especially if there is a free alternative that is the canonical choice. 😄

Serving a community of hundreds of thousands of developers is not cheap

EXCUSE ME?

I think people would be willing to support the infrostructure and actually pay for it (bintray did not fix buga and has not updated infrostructure for several years by the way, so some service would be nice).

say what?!

Still working on my conspiracy theory that they just want somebody to buy "jcenter". Like Google will announce that they're keeping it open read-only indefinitely (or whatever). We'll see.

Maybe some Redditors who got rich off of GME can buy it

I'm going to start reaching out to the big tech companies tomorrow to see if I can convince some PMs to buy the domain from JFrog and host a read-only version of JCenter. I have a few contacts I can leverage

That's probably 7 figures overhead a year just in bandwidth

Probably. And probably something Microsoft or Google could absorb. Or Cloudflare.

Reading the previous message, I would love to have a JetBrains repository, even if it's just links to JetBrains' Space for Ktor, etc. Especially since it's so easy to publish to Space. It would need compatibility with other platforms (GitLab...) but it would be really convenient, publishing to Maven Central is way too complicated (especially for multiplatform projects)

Hey, Package Search lead here: trying to think if there’s anything we can do to help out folks with the transition — we already index Central and maven.google.com (but not JCenter as they wouldn’t allow it), and are planning to add some other repos to our index too. We’ve got a new plugin major version in the works that will automatically add the right repositories for you when you add/update dependencies for which you have no repo in the build.

@seb please take a look at @irus solution here: https://github.com/Heapy/repo.kotlin.link. I've just tested it and it works perfectly. The idea is to make a distributed storage and use the central proxy only for querying paths.

Apart from that… let me know if you have feasible ideas (DM so we don’t derail this thread)

I would like to suggest that whatever solutions people think of are also applicable to the Java community. There are no reasons why this has to be Kotlin specific.

Maven is the best dependency management and distribution system so far. And it has all the tools for custom repositories we ever need. What we need is the decetralized way to host binaries with a centralized discovery. And even MavenCentral is a good thing. It guarantees that things that are hosted there, will be available forever, which also a good thing.

“Guarantees” … we hope. Apparently that guarantee wasn’t so strong for JCenter. Does anyone really need centralised coordinate mapping? Repos don’t provide discovery, the web does that. People find libraries by googling or finding them on GitHub, word of mouth, blogs etc. All coordinates/central repos are really doing is providing free bandwidth (until they don’t) and creating a not particularly well specified or useful extra namespace on top. If coordinates were just URLs to POM files instead of GAVs would anyone have ever cared?

Does anyone really need centralised coordinate mapping? Repos don’t provide discovery, the web does that. People find libraries by googling or finding them on GitHub, word of mouth, blogs etc.

That’s what we’re mostly doing at Package Search, too.

I have an answer for that 🙂 Imagine a modification to Maven/Gradle that did let you specify coordinates as URLs. OK, now we define a URL format that includes a hash. For example, it can in in the #fragment or the penultimate path segment (e.g.

<https://foo.com/86abfc9eb0dbdcdff706c331c29c8bf42eadc96827b0976d11ac3b90a9c60c2a.sha256/whatever.pom>

)

Maven specification allows that. But you still need to create a separate repository entry for each dependency. Including transitives.

shrug that’s OK. Projects need to find bandwidth anyway. GitHub pages for example can serve POMs and other small files, and if coordinates can be hash-URLs they can point to files on GH releases so the actual heavy lifting is done by the CDN. It means modifications to the build tools so they understand how to use this decentralised scheme. Because the root hash-URL you import into your build script contains the hash of the POM, and the POM itself contains the hash of the dependency POMS, you can get a tree of hashes that cover the entire dependency graph.

Good luck handling a gazillion separate repositories by hand in the build script 😛

The point of what I’m suggesting is that there wouldn’t be a repositories declaration anymore. If coordinates are hash-URLs managing the repositories block becomes superfluous. And yes, currently Maven/Gradle can’t support this. They require the centralised model. It’d mean some modifications to them, or an external tool to download dependencies (and then Maven/Gradle are just pointed at a directory of jars)

They’re always assumed to be in the same repository — but Gradle and Maven will happily resolve them from other declared repos, too, if need be

It’d be exactly as decentralised as the web is. Not more or less. The web is a success so it’s clearly not unrealistic, just different. My point is, the way the JVM ecosystem does things doesn’t make sense. We are totally exposed this kind of sudden shutdown, or worse, a compromise at a repository company. Which may never be detected and could already have happened. By switching to hash-URLs instead of GAVs the whole ecosystem becomes more robust, more decentralised and the tools become simpler too (managing repository blocks in Gradle and Maven is really a PITA). For example, the whole problem of “do you own the domain name for your groupID” would just disappear because there’d be no group IDs anymore. What justifies that additional complexity? Nothing really, it’s just the way things evolved.

The main difference with the web is that we’re not talking about cat pics, we’re talking about something that implies trust. Not just in terms of “am I talking to the right server”, but also “what they send me is the right stuff”. Imagine if, say, Amazon were served by decentralised servers. How do you prove the data they show is legit? It’s probably possible, but a huge problem space with an immense attack. surface. There’s an increasing degree of centralisation on the web, too, to guarantee trust

That’s the goal of the hashes. Say JB Package Search is a root of trust. Then you tell users the hash-URL for the POM of a library they find via your service. Once the user knows that URL, as long as it’s correct, every other server can be compromised, including any server hosting a transitive. It doesn’t matter - the hash is effectively a Merkle root over the dependency tree.

I agree the only hope for now for the Maven ecosystem is that, should Central run out of money, some big company could fund it back into existence (after all, I don’t think any big company doesn’t depend on the Maven ecosystem in some way)

The web has the same concept these days, it’s called Subresource Integrity. Same goal: it lets you use third party CDNs and block compromises, but in a decentralised way. You can add hashes to HTML tags that refer to external URLS.

But Package Search would then become a centralised source of truth. You’re just moving the problem around 🙂

Yeah, but PS is already a source of truth. You can serve us bad-but-plausible coordinates already, and we’re also exposed to hacked CDN roots or CDN mirrors. Also the above scheme doesn’t privilege package search. Honestly I get most of my coordinates from GitHub READMEs or documentation websites, and I find those in turn via a mix of sources.

Or someone fucked up in some way somewhere 😄

Same thing 😂

Ehehe tru dat

not yet 😃

I like the decentralised dependencies management idea 😉

Implement it and I’ll try to support it in Package Search 😉

Ha! You see! Success is inevitable! :blob-grin:

I said try 😛

So we can publish artifacts as torrents. Or blockchain. I hear blockchain is really good.

I love the idea 😁

Torrent is actually possible. But it still requires centralized dispatcher

inb4 using megaupload

Centralized dispatcher is bad? Do you want the network nodes to "vote" which version of the crypto library is the "right" one to send to link into your banking app?

As if banking development or any other security relevant development would use public repositories they don't control. 😄

I dont get it, why GitHub itself does not offering anonymous download of public packages. You can download the repo anonymous, the build artifacts anonymous, but not explicit public packages. Therefore you need a token... https://github.community/t/download-from-github-package-registry-without-authentication/14407/89

Probably to limit the traffic.

Probably just because not being done yet. It will be like that and already is for their docker registry. They just need ages to do it.

Not strictly Kotlin related, but hopefully relevant here: I wrote a new guide for publishing to MavenCentral (Android focused, should work outside of it too though, we use it for plain Kotlin modules as well) https://twitter.com/zsmb13/status/1357355313180082184

You should add the https://github.com/marcphilipp/nexus-publish-plugin/ plugin. Then you don't need to define the repos manually but have a nice helper for it and you will not have a problem when there are multiple open staging repositories, for example after an upload partly failed. Or the "scattered repos" you mention.

I was just complaining about there not being such guide earlier today 😀

Thank you @Vampire, I’ll check this out sometime!

I just went through the pain of publishing to sonatype with the new maven-publish plugin. Using the

io.codearte.nexus-staging

and

de.marcphilipp.nexus-publish

plugins made it so much easier than doing alot by hand

@seb I have no idea how Gradle internally implements dependencies, but from what I understand the previous proposal only touches the “I want to know where a pom file is”, not “what it is” or “how it's implemented”—it doesn't sound like it would be that difficult to add a

implementation(url("url-with-the-new-system", hash = "...", version = "..."))

that finds the pom file from the URL (and checks the hash, etc, without using the

repositories

block) and just feeds it to the next part of the internal pipeline. This way, the new system could get swapped-in as libraries expose a compatible coordinate with almost no changes to the configuration (and since it would allow anyone to publish to their own servers, it would be easier than jCenter or similar). The main downside would be that anyone with a Maven/Gradle version from before the system was implemented would not be able to pull a dependency that only has the new system, including transitive dependencies, but that's kind of the whole “you can slowly replace Java by Kotlin” thing, and as long as it's done slowly and libraries provide both schemes at first, should be doable. Group IDs wouldn't be coordinates anymore, but we can keep using them as documentation (much like Java packages are not a proof of ownership, but just a namespace), so apart from the “find the correct pom file”, almost all of Gradle wouldn't need to be modified. Another problem would be repos disappearing, but if there's project identification (via a hash, or a GPG signature, or whatever) it would be quite easy for anyone to copy the whole artifact on their own server, and any user could be sure it's the correct one (because the hash is the same).

Another problem would be repos disappearing, but if there's project identification (via a hash, or a GPG signature, or whatever) it would be quite easy for anyone to copy the whole artifact on their own server, and any user could be sure it's the correct one (because the hash is the same).

Which is a pretty big problem, as history builds are broken then. Of course builds using jcenter will also have broken history builds now, unless you use an init script or whatever to adjust the repository declaration. But with one or two big central repositories that are meant to stay, it is much more unlikely that this will happen. With using custom URLs, the risk that those go down is tremendously higher and experience shows that after 2-3 years many of them will be gone for good.

Right, ivy is a neat trick. If you’re worried about artifacts going AWOL then using a caching HTTP proxy should be sufficient, in theory? Someone could sell “caches as a service” if this is a common need and then you’re doing pure bandwidth reselling, with no trust required, so it should be cheap (i.e. anyone can offer that by just spinning up some VMs).

Then the cache goes down and you have the same problem. And it nees to be free for OSS projects.

I think most OSS websites need at least a website. Yeah, some just use a github readme, but many of them are just small and could easily just store the artifacts as binaries inside the git repo itself or something. Fundamentally, anything can disappear.

yeeeaahh ... no

Exactly. It doesn’t need anything fancy or complicated. Most of the support is there already. Programmatically searching Google is a problem unfortunately. They don’t make that easy, deliberately so. It’s not meant for programmatic access and they have a lot of tricks to shut such access down (I know, because I built some of them).

Ah, the famous "it doesn't sound like it would be that difficult"

Its a great guide, but also crazy that such a guide would be needed. Really artifact publishing should not be so complicated as this - clicking around in a 90's IE4 style UI should not be required, nor raising JIRA tickets, nor making xml in groovy dsl for a kotlin project.

GitHub packages already works, and I don't see why every CI system doesn't have a simple way to provide Maven-compatible artifact layout

GH Packages doesn’t provide anonymous access apparently (see above).

I'm fairly certain the GitLab equivalent is completely free & anonymous, however setting it up is a bit weird and it didn't manage it (but i'm also very far from being an expert, so that doesn't mean much)

I've managed to publish my first project to Central with your guide. Thanks @zsmb 🙂

Hosting a maven repository is stupidly easy. All you need is a web server and rsync. If you use gradle, you can actually deploy via scp, s3, gcs and probably a few other things. Great for internal projects (we push to a gcs bucket). And I have a repository on my website for an oss library. Otherwise jitpack.io and maven central are solid options. Jitpack is very easy to setup but does not support multi platform.

But JitPack is pretty unreliable and broken by design. No sane publishing should use it imho, but at most some temporary testing and for that I would actually use composite builds instead. 🙂

I thought Bintray was buggy ... but Sonatype Nexus is a nightmare! I'm trying to publish a tiny project with a few artifacts and I have a lot of timeouts from gradle and duplicate broken staging repos over and over again. And the whole process is so sloooow. I'm starting to wonder how I manage to publish KVision with more than 600 artifacts ... 😢

yesterday I published a lib and it was "closing" for over an hour. apparently, sually it should take a few minutes. so there's probably a lot more traffic right now

apparently there is something wrong https://status.maven.org/

To be fair, they are working hard on it https://twitter.com/Brian_Fox/status/1357414532512104448

Also to mention, they’re very likely to be experiencing way more traffic than usual these days. Hopefully it’ll go back to normal soon (which is, a bit buggy but not entirely broken)

@Robert Jaros

I thought Bintray was buggy ... but Sonatype Nexus is a nightmare! I'm trying to publish a tiny project with a few artifacts and I have a lot of timeouts from gradle and duplicate broken staging repos over and over again. And the whole process is so sloooow. I'm starting to wonder how I manage to publish KVision with more than 600 artifacts ...

It happens sometimes, that uploads are slow or are not working, but usually it works just fine. Probably mainly a problem of being flooded right now due to the announcement. If you use Gradle you should make sure to use the https://github.com/marcphilipp/nexus-publish-plugin/ which will explicitly create a staging repository up-front so that you don't have the problem of split staging repos and especially so if you use the https://github.com/Codearte/gradle-nexus-staging-plugin plugin to automate the repository closing that then also is told that staging repos id explicitly and does not need to use heuristics that depen on there being exactly one open staging repository.

is there any inside why they quit?

Only if you manage to get in touch with them directly. Hopefully they will have another blog post out next week with more details.

I see

Most likely: it costs money to run something like that. And now that JFrog is a public company, there likely was increased pressure to cut unjustifiable costs — I can’t imagine it was particularly effective as a funnel into their paid products.

The real question is: is it a effective enough for sonatype? :P

Iirc they don't operate MavenCentral, they just provide a place for OSS projects to release their projects to that is automatically synced to MavenCentral. Apache for example does not use it for their own projects but have an own sync, so the Sonatype part could anytime be replaced by someone else, or by bringing back the possibility to release to MavenCentral itself directly.

i just recently set up a maven repo that accepts upload via webdav, that is stupidly easy to set up in nginx

location /mavenupload/ {
    autoindex on;
    autoindex_exact_size off;
    autoindex_localtime on;
    alias /usr/share/nginx/data/maven/; # locaction of /maven on disk

    client_max_body_size 100M;
    dav_methods PUT DELETE MKCOL COPY MOVE;
    dav_access user:rw all:r;
    create_full_put_path on;

    auth_basic "Restricted";
    auth_basic_user_file /etc/nginx/.htpasswd;
}

location /maven/ {
    autoindex on;
    autoindex_exact_size off;
    autoindex_localtime on;
}

upload to

/mavenupload

and access

/maven

for packages but i wish github packages or something central, but easier to use than bintray/jcenter would exist i am not going to bother with mavencentral for a while myself

Of course a Maven repo is quickly setup. The problem is more that such quickly setup repos will typically not survive for too long and then again lead to broken builds. ;-)

i dunno if this has been flagged elsewhere, but it looks like we have a year to migrate off JCenter https://twitter.com/steveonjava/status/1358851394887770115

You still won’t be able to push new versions in a few months tho as far as I understand

Uploading ban has been pushed to March 31st. Lib authors definitely, but for us mere mortals it’s less of a panic

Yup

@zsmb forget the gradle-nexus-plugin I mentioned and also forget the gradle-nexus-staging-plugin. Szpak and Marc joined forces and combined both plugins into https://github.com/gradle-nexus/publish-plugin 🙂

I’ll take a look, thank you!