https://kotlinlang.org logo
Join Slack
Powered by
This is a joke, right? 3 months to migrate everyth...
# feed
c
This is a joke, right? 3 months to migrate everything off JCenter? How many Java/Kotlin libraries are now unmaintained and won’t get migrated? https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/
🤨 2
😱 32
I apologize that this link isn’t strictly-Kotlin, but it seemed very relevant for the Kotlin community
t
Thanks for that. I read it originally but it never clicked that it included jcenter
j
Yes, what is next since most of the Kotlin ecosystem is on there? And even if all new versions of libraries are released to someplace new it will likely mean a lot of older projects or going back in git history will break after May 1st.
The future for ktor and kotlin releases seems to be https://maven.pkg.jetbrains.space/
a
It is is really terrible. Does anybody have any ideas. Publishing to maven central is a pain. I am not sure it is possible to do without controlling top level domain with the correct name
r
First TravisCI, now Bintray ... what's going on with this world?!
a
@Jurriaan Mous sadly Space does not provide a single repository even for a single space. Repositories are per-project
c
TravisCI at least has some decent replacements, and ultimately only impacted your own project. If my dependencies don’t get migrated, I can’t migrate my own projects’ builds
Are there any services that mirror Maven repos and cache artifacts indefinitely?
j
@altavir Ah ok, I am not yet familiar with space but I noticed the Ktor EAP releases where published here: https://maven.pkg.jetbrains.space/public/p/ktor/eap
a
We are hosting some internal builds in our Space, but as I said, it has the same problem as with github, you have to write a repository for each project and you have to write them in all dependencies
v
Publishing to Maven Central is not really problematic. The only "hard" part is the initial getting account and being set up. But after that, it just works usually.
➕ 5
🚫 4
c
It’s problematic in that you need to own the domain for the groupId you’re publishing under, while JCenter did not have that restriction. I’m sure a lot of libraries would need to change their artifact coordinates to be allowed to publish to MavenCentral
j
I'm concerned about the security implications of all of this. There are some serious risks associated with typo-squatting. And if Central suddenly gets a huge influx of requests for new packages, that's going to seriously overwhelm them.
v
You don't need to own the domain. There are for example also 
com.github.<user>
libraries published on Maven Central
a
our library currently do not follow domain naming at all. We just use kscience prefix. And I am not sure that I want to change the naming just to satisfy maven-central draconic rules.
👆 2
v
Where does it say you need that?
People using 
com.github.whatever
also don't own that domain
In the guide it is mentioned because it is a good convention to prevent name-clashes
a
OK, I will check that, thank you
c
Not necessarily owning the actual TLD, but it does require ownership of the domain or subdomain.
you must choose a groupId for a domain that you own, or for which you are the designated maintainer on behalf of the owner
https://central.sonatype.org/pages/producers.html#individual-projects-open-source-software-repository-hosting-ossrh
j
The security risks here around typosquatting and namespace hijacking are real. https://blog.autsoft.hu/a-confusing-dependency/
👍 1
c
But like @altavir said, I don’t want to have to change to a 
io.github.whatever
groupId. It’s long, ugly, and would be a breaking change for the consumers of the dependency if it wasn’t using that already
v
Hm, I forgot I had to prove I own the domain. Actually it is a bit senseless too, as you can anytime give up the domain or change your GitHub username after you got the namespace assigned. Well, we will see whether Sonatype comes up with some changes in reaction.
l
Versions submited to Maven Central can never be replaced tho
So if you're using a dependency from there and don't change the version, it won't be a security risk
v
Versions submited to Maven Central can never be replaced tho
Which usually is a good thing imho 🙂
☝️ 1
l
I think so too! But I just wanted to ensure that if, for example, Maven Central decides to clone JCenter, it won't be a security risk in that sense
m
The JetBrains could create a Maven Repo for the Community Packages 🤔
👍 2
a
It is not cheap to maintain. We just discussed it in telegram and it seems that its much easier to create virtual maven repository that maps requests to small github/space/whatever repositories. It could be based on https://package-search.jetbrains.com/. It won't be quite safe, but it would ensure that all artifacts are resolved with a single repository without a lot of traffic through the server.
m
I see you point, but I think a oficial Community Repo from JetBrains will help a lot the Kotlin Ecosystem to grow
👍 2
a
Indeed, but it could be too expensive
v
Especially if there is a free alternative that is the canonical choice. 😄
➕ 2
k
Serving a community of hundreds of thousands of developers is not cheap
s
EXCUSE ME?
😆 2
Wow, this makes me incredibly angry
a
I think people would be willing to support the infrostructure and actually pay for it (bintray did not fix buga and has not updated infrostructure for several years by the way, so some service would be nice).
p
say what?!
k
Still working on my conspiracy theory that they just want somebody to buy "jcenter". Like Google will announce that they're keeping it open read-only indefinitely (or whatever). We'll see.
k
Maybe some Redditors who got rich off of GME can buy it
😄 12
j
I'm going to start reaching out to the big tech companies tomorrow to see if I can convince some PMs to buy the domain from JFrog and host a read-only version of JCenter. I have a few contacts I can leverage
k
That's probably 7 figures overhead a year just in bandwidth
The domain name would be peanuts
j
Probably. And probably something Microsoft or Google could absorb. Or Cloudflare.
c
Reading the previous message, I would love to have a JetBrains repository, even if it's just links to JetBrains' Space for Ktor, etc. Especially since it's so easy to publish to Space. It would need compatibility with other platforms (GitLab...) but it would be really convenient, publishing to Maven Central is way too complicated (especially for multiplatform projects)
s
Hey, Package Search lead here: trying to think if there’s anything we can do to help out folks with the transition — we already index Central and maven.google.com (but not JCenter as they wouldn’t allow it), and are planning to add some other repos to our index too. We’ve got a new plugin major version in the works that will automatically add the right repositories for you when you add/update dependencies for which you have no repo in the build.
a
@seb please take a look at @irus solution here: https://github.com/Heapy/repo.kotlin.link. I've just tested it and it works perfectly. The idea is to make a distributed storage and use the central proxy only for querying paths.
➕ 1
s
Apart from that… let me know if you have feasible ideas (DM so we don’t derail this thread)
m
I would like to suggest that whatever solutions people think of are also applicable to the Java community. There are no reasons why this has to be Kotlin specific.
☝️ 3
Honestly, this might be a good time for the whole JVM community to rethink Maven repositories as a system. Is there any specific reason we all have to get our artifacts from a few central places? Every open source project can publish GitHub releases or host websites. It’s not obvious to me that the whole repository+coordinates structure has ever been optimal, to be honest.
a
Maven is the best dependency management and distribution system so far. And it has all the tools for custom repositories we ever need. What we need is the decetralized way to host binaries with a centralized discovery. And even MavenCentral is a good thing. It guarantees that things that are hosted there, will be available forever, which also a good thing.
💯 5
m
“Guarantees” … we hope. Apparently that guarantee wasn’t so strong for JCenter. Does anyone really need centralised coordinate mapping? Repos don’t provide discovery, the web does that. People find libraries by googling or finding them on GitHub, word of mouth, blogs etc. All coordinates/central repos are really doing is providing free bandwidth (until they don’t) and creating a not particularly well specified or useful extra namespace on top. If coordinates were just URLs to POM files instead of GAVs would anyone have ever cared?
s
Does anyone really need centralised coordinate mapping? Repos don’t provide discovery, the web does that. People find libraries by googling or finding them on GitHub, word of mouth, blogs etc.
That’s what we’re mostly doing at Package Search, too.
I would not trust putting random URLs in my build, to be honest. Who’s to guarantee they’re not malicious? That’s the big plus of centralisation: you know the host isn’t malicious. And it’s relatively difficult to put malware in Central, while posing as someone else, as you first need to compromise an account with the rights to publish
👍 5
m
I have an answer for that 🙂 Imagine a modification to Maven/Gradle that did let you specify coordinates as URLs. OK, now we define a URL format that includes a hash. For example, it can in in the #fragment or the penultimate path segment (e.g. 
<https://foo.com/86abfc9eb0dbdcdff706c331c29c8bf42eadc96827b0976d11ac3b90a9c60c2a.sha256/whatever.pom>
)
The HTTP client understands that when the URL matches that format, the content it receives must match the hash. Now project READMEs that include coordinates just provide the URL instead of a 
com.foo:bar:1.2.3
string. The URL becomes self-authenticating. This is better because the project docs are the root of trust anyway.
(i.e. if someone compromises the project account, they don’t actually need to hack Maven Central to get new users, they can just change the coordinate to one that looks plausible but is wrong)
a
Maven specification allows that. But you still need to create a separate repository entry for each dependency. Including transitives.
☝️ 2
m
shrug that’s OK. Projects need to find bandwidth anyway. GitHub pages for example can serve POMs and other small files, and if coordinates can be hash-URLs they can point to files on GH releases so the actual heavy lifting is done by the CDN. It means modifications to the build tools so they understand how to use this decentralised scheme. Because the root hash-URL you import into your build script contains the hash of the POM, and the POM itself contains the hash of the dependency POMS, you can get a tree of hashes that cover the entire dependency graph.
When hash-URLs are used, it means you don’t even need SSL on the CDNs at that point so artifacts can be cached by normal or transparent HTTP proxies. Good for corporate networks. Instead of special “repository managers” you can just deploy a regular HTTP proxy with a big cache, point build tools to it, and you’re done.
s
Good luck handling a gazillion separate repositories by hand in the build script 😛
Gradle and Maven will not resolve automatically dependencies outside the repositories you declare, for obvious reasons — and that goes for transitive repositories too. If every transitive dependency can be on any repository anywhere, managing the repositories declaration becomes borderline impossible
AFAIK there is no way to declare where transitive dependencies are located, in poms.
m
The point of what I’m suggesting is that there wouldn’t be a repositories declaration anymore. If coordinates are hash-URLs managing the repositories block becomes superfluous. And yes, currently Maven/Gradle can’t support this. They require the centralised model. It’d mean some modifications to them, or an external tool to download dependencies (and then Maven/Gradle are just pointed at a directory of jars)
👍 1
s
They’re always assumed to be in the same repository — but Gradle and Maven will happily resolve them from other declared repos, too, if need be
So essentially you’d want an entirely new ecosystem, decentralised-but-not-really (because you still need discovery, and hard links to other repositories for transitive dependencies)? Sounds a tad unrealistic when you consider the implications
Especially when there’s no guarantees about the various repositories you may need to get dependencies from — not just security-wise (hashes can have conflicts), but also in terms of whether they’ll exist when you run the build. If they don’t, how’s the build system supposed to find a dependency in a distributed system?
m
It’d be exactly as decentralised as the web is. Not more or less. The web is a success so it’s clearly not unrealistic, just different. My point is, the way the JVM ecosystem does things doesn’t make sense. We are totally exposed this kind of sudden shutdown, or worse, a compromise at a repository company. Which may never be detected and could already have happened. By switching to hash-URLs instead of GAVs the whole ecosystem becomes more robust, more decentralised and the tools become simpler too (managing repository blocks in Gradle and Maven is really a PITA). For example, the whole problem of “do you own the domain name for your groupID” would just disappear because there’d be no group IDs anymore. What justifies that additional complexity? Nothing really, it’s just the way things evolved.
The problem of artifacts disappearing is indeed real but, we already face it (hence this thread). Projects can cache and re-host their transitive dependencies, nothing hard about that. SHA2 hashes don’t have conflicts, or at least if you find one please show me so I can steal it and become famous 😂
Anyway, just brainstorming. I agree obviously such a change can’t be done globally in 3 months. But, consider the worst case scenario - Sonatype can’t sustain Maven Central just like JFrog couldn’t sustain JCenter. Then we’re all hosed and would have no choice (or hope a rich company bails the community out because we’re too big to fail)
s
The main difference with the web is that we’re not talking about cat pics, we’re talking about something that implies trust. Not just in terms of “am I talking to the right server”, but also “what they send me is the right stuff”. Imagine if, say, Amazon were served by decentralised servers. How do you prove the data they show is legit? It’s probably possible, but a huge problem space with an immense attack. surface. There’s an increasing degree of centralisation on the web, too, to guarantee trust
m
That’s the goal of the hashes. Say JB Package Search is a root of trust. Then you tell users the hash-URL for the POM of a library they find via your service. Once the user knows that URL, as long as it’s correct, every other server can be compromised, including any server hosting a transitive. It doesn’t matter - the hash is effectively a Merkle root over the dependency tree.
s
I agree the only hope for now for the Maven ecosystem is that, should Central run out of money, some big company could fund it back into existence (after all, I don’t think any big company doesn’t depend on the Maven ecosystem in some way)
m
The web has the same concept these days, it’s called Subresource Integrity. Same goal: it lets you use third party CDNs and block compromises, but in a decentralised way. You can add hashes to HTML tags that refer to external URLS.
yes black 1
s
But Package Search would then become a centralised source of truth. You’re just moving the problem around 🙂
If your answer is “have more sources of truth”, what if they disagree? That’s something subresource integrity doesn’t protect from.
m
Yeah, but PS is already a source of truth. You can serve us bad-but-plausible coordinates already, and we’re also exposed to hacked CDN roots or CDN mirrors. Also the above scheme doesn’t privilege package search. Honestly I get most of my coordinates from GitHub READMEs or documentation websites, and I find those in turn via a mix of sources.
If they disagree it means someone got hacked. Useful to know 🙂
s
Or someone fucked up in some way somewhere 😄
m
Same thing 😂
s
Ehehe tru dat
I don’t think there’s enough momentum in the ecosystem to actually move away from the Maven model, for now — although JCenter’s imminent demise sure has been a wake up call for many
It’s always good to discuss such ideas (it also lets me play devil’s advocate 😄), but it won’t necessarily get anything done. AFAIK no package management system out there is truly decentralised
m
not yet 😃
m
I like the decentralised dependencies management idea 😉
s
Implement it and I’ll try to support it in Package Search 😉
m
Ha! You see! Success is inevitable! blob grin
s
I said try 😛
😂 4
Y’all need to get intellij to support your build system first 😄
or write a plugin to do it yourselves
right, time to go back to figuring out a bug that’s been stumping me for a few days
👋
k
So we can publish artifacts as torrents. Or blockchain. I hear blockchain is really good.
💯 3
😂 4
😆 1
m
I love the idea 😁
a
Torrent is actually possible. But it still requires centralized dispatcher
s
inb4 using megaupload
k
Centralized dispatcher is bad? Do you want the network nodes to "vote" which version of the crypto library is the "right" one to send to link into your banking app?
v
As if banking development or any other security relevant development would use public repositories they don't control. 😄
h
I dont get it, why GitHub itself does not offering anonymous download of public packages. You can download the repo anonymous, the build artifacts anonymous, but not explicit public packages. Therefore you need a token... https://github.community/t/download-from-github-package-registry-without-authentication/14407/89
a
Probably to limit the traffic.
☝️ 4
v
Probably just because not being done yet. It will be like that and already is for their docker registry. They just need ages to do it.
z
Not strictly Kotlin related, but hopefully relevant here: I wrote a new guide for publishing to MavenCentral (Android focused, should work outside of it too though, we use it for plain Kotlin modules as well) https://twitter.com/zsmb13/status/1357355313180082184
🎉 8
➕ 4
👏 9
v
You should add the https://github.com/marcphilipp/nexus-publish-plugin/ plugin. Then you don't need to define the repos manually but have a nice helper for it and you will not have a problem when there are multiple open staging repositories, for example after an upload partly failed. Or the "scattered repos" you mention.
b
I was just complaining about there not being such guide earlier today 😀
z
Thank you @Vampire, I’ll check this out sometime!
t
I just went through the pain of publishing to sonatype with the new maven-publish plugin. Using the 
io.codearte.nexus-staging
and 
de.marcphilipp.nexus-publish
plugins made it so much easier than doing alot by hand
c
@seb I have no idea how Gradle internally implements dependencies, but from what I understand the previous proposal only touches the “I want to know where a pom file is”, not “what it is” or “how it's implemented”—it doesn't sound like it would be that difficult to add a 
implementation(url("url-with-the-new-system", hash = "...", version = "..."))
that finds the pom file from the URL (and checks the hash, etc, without using the 
repositories
block) and just feeds it to the next part of the internal pipeline. This way, the new system could get swapped-in as libraries expose a compatible coordinate with almost no changes to the configuration (and since it would allow anyone to publish to their own servers, it would be easier than jCenter or similar). The main downside would be that anyone with a Maven/Gradle version from before the system was implemented would not be able to pull a dependency that only has the new system, including transitive dependencies, but that's kind of the whole “you can slowly replace Java by Kotlin” thing, and as long as it's done slowly and libraries provide both schemes at first, should be doable. Group IDs wouldn't be coordinates anymore, but we can keep using them as documentation (much like Java packages are not a proof of ownership, but just a namespace), so apart from the “find the correct pom file”, almost all of Gradle wouldn't need to be modified. Another problem would be repos disappearing, but if there's project identification (via a hash, or a GPG signature, or whatever) it would be quite easy for anyone to copy the whole artifact on their own server, and any user could be sure it's the correct one (because the hash is the same).
v
Another problem would be repos disappearing, but if there's project identification (via a hash, or a GPG signature, or whatever) it would be quite easy for anyone to copy the whole artifact on their own server, and any user could be sure it's the correct one (because the hash is the same).
Which is a pretty big problem, as history builds are broken then. Of course builds using jcenter will also have broken history builds now, unless you use an init script or whatever to adjust the repository declaration. But with one or two big central repositories that are meant to stay, it is much more unlikely that this will happen. With using custom URLs, the risk that those go down is tremendously higher and experience shows that after 2-3 years many of them will be gone for good.
Btw. you can already use arbitrary URLs for your dependencies. Just define an 
ivy
repository with a custom pattern layout. You can even configure it to not look for metadata but to just use the 
jar
that was found if that is what you want.
m
Right, ivy is a neat trick. If you’re worried about artifacts going AWOL then using a caching HTTP proxy should be sufficient, in theory? Someone could sell “caches as a service” if this is a common need and then you’re doing pure bandwidth reselling, with no trust required, so it should be cheap (i.e. anyone can offer that by just spinning up some VMs).
v
Then the cache goes down and you have the same problem. And it nees to be free for OSS projects.
But actually, why not just do a Google search for the coordinates programmatically and use the first search result, then parse the site for the link to a POM and use that as dependency. That should be future-proof as long as Google does not vanish.
m
I think most OSS websites need at least a website. Yeah, some just use a github readme, but many of them are just small and could easily just store the artifacts as binaries inside the git repo itself or something. Fundamentally, anything can disappear.
Real simple - just create a zip with all the dependency artifacts in it, like a fat JAR would be. Then store it alongside each release. If the servers disappear, users can just download the zip and extract it to a libs directory, which the system would scan first as if it were a local http cache i.e. if the file is present and the hash matches what the URL would be, the download is skipped. These sorts of simple solutions can work well enough most of the time, I think.
v
yeeeaahh ... no
Besides that you also can use a 
flatDir
repository already
m
Exactly. It doesn’t need anything fancy or complicated. Most of the support is there already. Programmatically searching Google is a problem unfortunately. They don’t make that easy, deliberately so. It’s not meant for programmatic access and they have a lot of tricks to shut such access down (I know, because I built some of them).
k
Ah, the famous "it doesn't sound like it would be that difficult"
☝️ 1
yes black 2
😂 4
j
Its a great guide, but also crazy that such a guide would be needed. Really artifact publishing should not be so complicated as this - clicking around in a 90's IE4 style UI should not be required, nor raising JIRA tickets, nor making xml in groovy dsl for a kotlin project.
👍 2
☝️ 2
e
GitHub packages already works, and I don't see why every CI system doesn't have a simple way to provide Maven-compatible artifact layout
m
GH Packages doesn’t provide anonymous access apparently (see above).
c
I'm fairly certain the GitLab equivalent is completely free & anonymous, however setting it up is a bit weird and it didn't manage it (but i'm also very far from being an expert, so that doesn't mean much)
r
I've managed to publish my first project to Central with your guide. Thanks @zsmb 🙂
🎉 1
❤️ 1
j
Hosting a maven repository is stupidly easy. All you need is a web server and rsync. If you use gradle, you can actually deploy via scp, s3, gcs and probably a few other things. Great for internal projects (we push to a gcs bucket). And I have a repository on my website for an oss library. Otherwise jitpack.io and maven central are solid options. Jitpack is very easy to setup but does not support multi platform.
v
But JitPack is pretty unreliable and broken by design. No sane publishing should use it imho, but at most some temporary testing and for that I would actually use composite builds instead. 🙂
💯 3
r
I thought Bintray was buggy ... but Sonatype Nexus is a nightmare! I'm trying to publish a tiny project with a few artifacts and I have a lot of timeouts from gradle and duplicate broken staging repos over and over again. And the whole process is so sloooow. I'm starting to wonder how I manage to publish KVision with more than 600 artifacts ... 😢
Is it always like this or is this just the effect of massive bintray to central migration?
m
yesterday I published a lib and it was "closing" for over an hour. apparently, sually it should take a few minutes. so there's probably a lot more traffic right now
r
apparently there is something wrong https://status.maven.org/
f

https://youtu.be/cc8f639H6dQ▾

z
To be fair, they are working hard on it https://twitter.com/Brian_Fox/status/1357414532512104448
s
Also to mention, they’re very likely to be experiencing way more traffic than usual these days. Hopefully it’ll go back to normal soon (which is, a bit buggy but not entirely broken)
v
@Robert Jaros
I thought Bintray was buggy ... but Sonatype Nexus is a nightmare! I'm trying to publish a tiny project with a few artifacts and I have a lot of timeouts from gradle and duplicate broken staging repos over and over again. And the whole process is so sloooow. I'm starting to wonder how I manage to publish KVision with more than 600 artifacts ...
It happens sometimes, that uploads are slow or are not working, but usually it works just fine. Probably mainly a problem of being flooded right now due to the announcement. If you use Gradle you should make sure to use the https://github.com/marcphilipp/nexus-publish-plugin/ which will explicitly create a staging repository up-front so that you don't have the problem of split staging repos and especially so if you use the https://github.com/Codearte/gradle-nexus-staging-plugin plugin to automate the repository closing that then also is told that staging repos id explicitly and does not need to use heuristics that depen on there being exactly one open staging repository.
s
is there any inside why they quit?
j
Only if you manage to get in touch with them directly. Hopefully they will have another blog post out next week with more details.
s
I see
I started only 12 months ago to use it now have to migrate 10 projects, doh.
s
Most likely: it costs money to run something like that. And now that JFrog is a public company, there likely was increased pressure to cut unjustifiable costs — I can’t imagine it was particularly effective as a funnel into their paid products.
m
The real question is: is it a effective enough for sonatype? :P
v
Iirc they don't operate MavenCentral, they just provide a place for OSS projects to release their projects to that is automatically synced to MavenCentral. Apache for example does not use it for their own projects but have an own sync, so the Sonatype part could anytime be replaced by someone else, or by bringing back the possibility to release to MavenCentral itself directly.
n
i just recently set up a maven repo that accepts upload via webdav, that is stupidly easy to set up in nginx
Copy code
location /mavenupload/ {
    autoindex on;
    autoindex_exact_size off;
    autoindex_localtime on;
    alias /usr/share/nginx/data/maven/; # locaction of /maven on disk

    client_max_body_size 100M;
    dav_methods PUT DELETE MKCOL COPY MOVE;
    dav_access user:rw all:r;
    create_full_put_path on;

    auth_basic "Restricted";
    auth_basic_user_file /etc/nginx/.htpasswd;
}

location /maven/ {
    autoindex on;
    autoindex_exact_size off;
    autoindex_localtime on;
}
upload to 
/mavenupload
and access 
/maven
for packages but i wish github packages or something central, but easier to use than bintray/jcenter would exist i am not going to bother with mavencentral for a while myself
v
Of course a Maven repo is quickly setup. The problem is more that such quickly setup repos will typically not survive for too long and then again lead to broken builds. ;-)
☝️ 6
t
i dunno if this has been flagged elsewhere, but it looks like we have a year to migrate off JCenter https://twitter.com/steveonjava/status/1358851394887770115
➕ 1
s
You still won’t be able to push new versions in a few months tho as far as I understand
Just download
So it’s still on lib authors to migrate to Central ASAP
t
Uploading ban has been pushed to March 31st. Lib authors definitely, but for us mere mortals it’s less of a panic
s
Yup
f
v
@zsmb forget the gradle-nexus-plugin I mentioned and also forget the gradle-nexus-staging-plugin. Szpak and Marc joined forces and combined both plugins into https://github.com/gradle-nexus/publish-plugin 🙂
➕ 1
♥️ 2
z
I’ll take a look, thank you!
12 Views
Open in Slack
PreviousNext
Powered by