Hello, I'm advertising the #library-development channel that we have on this Slack. We can use it to chat about our pain points and solutions as library developers (like overcoming the 405/409 http errors from bintray that make publishing workflows flaky 😬). If you make libraries or plan to do it, consider joining that channel!

Ooooh, is there a discussion about this already? I'm super interested in getting these to work reliably.

@mbonnin I was discussing in DM with @Arkadii Ivanov since I saw we had the same issue, and in the meantime, I discovered that having

publish=1

when uploading to bintray is the culprit that triggers unreliability like HTTP 405&409 errors. The good news is that you can trigger publishing after all of that with a single http call https://bintray.com/docs/api/#_publish_discard_uploaded_content

Yep. This flag should be disabled. And also having not many concurrent uploads and upload all metadata in a separate job.

Damn, why not?

To me it looks like bintray don't like concurrent uploads. Also there could be same files uploaded from different jobs, not sure. But it will significantly reduce flakiness

BTW, it's possible that there's also network unreliability on GitHub Actions that would be the cause of the timeouts. I'll implement what I was talking about this weekend and see the results. If there's network issues when uploading from GitHub Actions, I'll limit it to publish Windows and Linux exclusive targets, and publish the rest locally from my Mac.

Also successful upload does not really mean all required files are there. I had issues like metadata files not uploading after gradle upgrade, or JS artifacts corrupted for unknown reason. So verification step is definitely required before publication.

When you say verification, it's manual verification, right? There's no process like sonatype's "close" process that checks the artifacts?

@mbonnin He means having a project that adds the bintray repo with credentials to see unpublished content, and that depends on all the modules there for all supported targets. Having this step run after upload and before publishing allows automatic verification.

Yep https://github.com/arkivanov/MVIKotlin/blob/master/tools/check-publication/build.gradle.kts

I see, thanks. That's more involved than what I hoped for but good to know 👍

Its job https://github.com/arkivanov/MVIKotlin/blob/6eff7f2e5e9d08aaf9e3024bc45e11574c7e2431/.github/workflows/publish.yml#L111

Tentative fix there: https://github.com/apollographql/apollo-android/commit/660e39845f2db60fd57b1b0016b32e582584d250

Don't you need the

override

flag still?

Do I?

@mbonnin You might still need

override=1

in case you have to retry the job because of a network issue.

Ah yes, if there's a network issue, override is still required

Unless you don't have concurrent jobs. Otherwise same files can be uploaded concurrently.

Yea, I only have one job, that's easier

Also, you leak the OkHttp connection (misses

use { … }

or a call to

close

, though it doesn't natter much since the VM should be shutdown afterwards.

Did we reach out to bintray about all this?

Then should be ok. And even not flaky.

We didn't reach bintray, but we can spread the word about our findings once we can confirm all that.

Then should be ok. And even not flaky

I've definitely seen my fair share of 409/405 (and also 443). I also used to run gradle with

--parallel

so maybe that's another reason

Please share their response in case 😀

@mbonnin 443 wasn't ah http error status code but the port of https where the connection failed at some point (often timeout, the cause might be GitHub: https://github.com/actions/virtual-environments/issues/1187)

😬

This adds-up to GitHub servers and networking costs, I'll keep on retrying until it succeeds. If it keeps on failing often, I might automate retries, or switch to another CI provider.

I see

Yes, that's what I'm doing. But I'll wait doing something else, like complaining here 😁

Ahah, I was going to say, this costs me attention time too, I don't like doing something else while a release is ongoing

Until we get sub-minute build and release times, it's always worth doing something else during the process.

Also, slightly related, did you benchmark upload speed of bintray vs sonatype?

Nope, I'm not uploading to sonatype (yet?), but since I disabled publish on every upload, it's no longer a bottleneck. The publishing after upload took less than 8 minutes, but I have so many artifacts (I'll try to tell you how many if I can catch the CI mid-success) that I'm not surprised nor bothered by it.

1688 😮

Hallelujah! After disabling Gradle's configure-on-demand and using bash on Windows (instead of default powershell that parses dots differently), the whole process works wonder! I'm so happy! Thank you so much to both of you @Arkadii Ivanov and @mbonnin for the inspiration, the help (including in DM), support and all. Here's the CI run. The most interesting is the last one which shows the publish API call retries at play. https://github.com/LouisCAD/Splitties/runs/1052298465?check_suite_focus=true