Hello, I'm advertising the <#C8C4JTXR7|library-development> channel that we have on this Slack. We c...

# multiplatform

[louiscad]

Hello, I'm advertising the #library-development channel that we have on this Slack.
We can use it to chat about our pain points and solutions as library developers (like overcoming the 405/409 http errors from bintray that make publishing workflows flaky 😬).

If you make libraries or plan to do it, consider joining that channel!

[mbonnin]

Ooooh, is there a discussion about this already? I'm super interested in getting these to work reliably.

[louiscad]

@mbonnin I was discussing in DM with @Arkadii Ivanov since I saw we had the same issue, and in the meantime, I discovered that having publish=1 when uploading to bintray is the culprit that triggers unreliability like HTTP 405&409 errors. The good news is that you can trigger publishing after all of that with a single http call https://bintray.com/docs/api/#_publish_discard_uploaded_content

[Arkadii Ivanov]

Yep. This flag should be disabled. And also having not many concurrent uploads and upload all metadata in a separate job.

[Arkadii Ivanov]

But it won't completely fix the problem

[mbonnin]

Damn, why not?

[Arkadii Ivanov]

To me it looks like bintray don't like concurrent uploads. Also there could be same files uploaded from different jobs, not sure. But it will significantly reduce flakiness

[louiscad]

BTW, it's possible that there's also network unreliability on GitHub Actions that would be the cause of the timeouts. I'll implement what I was talking about this weekend and see the results. If there's network issues when uploading from GitHub Actions, I'll limit it to publish Windows and Linux exclusive targets, and publish the rest locally from my Mac.

[louiscad]

It really worked wonders in parallel without the publish flag enabled, I think it's more concurrent publishing that it doesn't like.

[Arkadii Ivanov]

Also successful upload does not really mean all required files are there. I had issues like metadata files not uploading after gradle upgrade, or JS artifacts corrupted for unknown reason. So verification step is definitely required before publication.

[mbonnin]

When you say verification, it's manual verification, right? There's no process like sonatype's "close" process that checks the artifacts?

[mbonnin]

(which is a bummer because with mpp it's hard to keep track of all the artifacts...)

[louiscad]

@mbonnin He means having a project that adds the bintray repo with credentials to see unpublished content, and that depends on all the modules there for all supported targets. Having this step run after upload and before publishing allows automatic verification.

[Arkadii Ivanov]

Yep https://github.com/arkivanov/MVIKotlin/blob/master/tools/check-publication/build.gradle.kts

[Arkadii Ivanov]

It enabled only when a flag is passed https://github.com/arkivanov/MVIKotlin/blob/6eff7f2e5e9d08aaf9e3024bc45e11574c7e2431/settings.gradle.kts#L30

[mbonnin]

I see, thanks. That's more involved than what I hoped for but good to know 👍

[Arkadii Ivanov]

Its job https://github.com/arkivanov/MVIKotlin/blob/6eff7f2e5e9d08aaf9e3024bc45e11574c7e2431/.github/workflows/publish.yml#L111

[mbonnin]

Tentative fix there: https://github.com/apollographql/apollo-android/commit/660e39845f2db60fd57b1b0016b32e582584d250

[mbonnin]

Obviously not tested because this requires a release but I'll test for next one

[mbonnin]

we only run a macOS job at the moment (no windows) so it's easy to run everything in the same job

[Arkadii Ivanov]

Don't you need the override flag still?

[mbonnin]

Do I?

[louiscad]

@mbonnin You might still need override=1 in case you have to retry the job because of a network issue.

[mbonnin]

Ah yes, if there's a network issue, override is still required

[mbonnin]

or discard-retry is another option, this is how sonatype would do

[Arkadii Ivanov]

Unless you don't have concurrent jobs. Otherwise same files can be uploaded concurrently.

[mbonnin]

Yea, I only have one job, that's easier

[louiscad]

Also, you leak the OkHttp connection (misses use { … } or a call to close, though it doesn't natter much since the VM should be shutdown afterwards.

[mbonnin]

Did we reach out to bintray about all this?

[Arkadii Ivanov]

Then should be ok. And even not flaky.

[louiscad]

We didn't reach bintray, but we can spread the word about our findings once we can confirm all that.

[mbonnin]

Then should be ok. And even not flaky

I've definitely seen my fair share of 409/405 (and also 443). I also used to run gradle with --parallel so maybe that's another reason

[mbonnin]

If next release fails, I'll reach out to bintray, they usually respond quite fast

[Arkadii Ivanov]

Please share their response in case 😀

[louiscad]

@mbonnin 443 wasn't ah http error status code but the port of https where the connection failed at some point (often timeout, the cause might be GitHub: https://github.com/actions/virtual-environments/issues/1187)

[louiscad]

@mbonnin I'm following up, here's the commit where I add a new workflow that includes uploaded dependencies check as @Arkadii Ivanov suggested, and finally publishes via Bintray's API, with retries for up to an hour. https://github.com/LouisCAD/Splitties/commit/e0f8c596e3d738316bcb720b9ef3062bf9caa7cf

That commit just misses the .gitignore file for the new module which I added just after here 😅: https://github.com/LouisCAD/Splitties/commit/0125b2bdbca70c2e54ed4551da82e2fc932cbfa7

[louiscad]

… and it failed because of GitHub Actions unreliable network…

[louiscad]

But no 405 or 409 http from Bintray, and it worked flawlessly locally.

[mbonnin]

😬

[mbonnin]

So in these cases, you're relaunching the GA job?

[mbonnin]

And waiting 20minutes again?

[louiscad]

This adds-up to GitHub servers and networking costs, I'll keep on retrying until it succeeds. If it keeps on failing often, I might automate retries, or switch to another CI provider.

[mbonnin]

I see

[louiscad]

Yes, that's what I'm doing. But I'll wait doing something else, like complaining here 😁

[mbonnin]

Ahah, I was going to say, this costs me attention time too, I don't like doing something else while a release is ongoing

[mbonnin]

If it takes hours, I prefer doing it manually for the time being

[louiscad]

Until we get sub-minute build and release times, it's always worth doing something else during the process.

[mbonnin]

Also, slightly related, did you benchmark upload speed of bintray vs sonatype?

[mbonnin]

I'm under the impression that bintray is significantly slower but only have anecdotal evidence

[louiscad]

Nope, I'm not uploading to sonatype (yet?), but since I disabled publish on every upload, it's no longer a bottleneck. The publishing after upload took less than 8 minutes, but I have so many artifacts (I'll try to tell you how many if I can catch the CI mid-success) that I'm not surprised nor bothered by it.

[louiscad]

I have 1688 files uploaded for the current dev version of Splitties.

Currently praying for the publication check to not fail again because of networking issue, as I'm writing a ticket for them to consider.

[mbonnin]

1688 😮

[louiscad]

Hallelujah!

https://www.youtube.com/watch?v=beWJ_MdvyLQ▾

After disabling Gradle's configure-on-demand and using bash on Windows (instead of default powershell that parses dots differently), the whole process works wonder!

I'm so happy! Thank you so much to both of you @Arkadii Ivanov and @mbonnin for the inspiration, the help (including in DM), support and all.

Here's the CI run. The most interesting is the last one which shows the publish API call retries at play.
https://github.com/LouisCAD/Splitties/runs/1052298465?check_suite_focus=true