Hi, this is kind of a repeat of last saturdays mes...

# arrow-contributors

[Jannis]

Hi, this is kind of a repeat of last saturdays message: There are a few more problems currently present because of the repo split that need to be addressed:
Problems currently (That I can think of):
• it is impossible to predict where changes break code. Even for someone who knows almost the entire arrow code base it is hard to predict. This means it is literally impossible for new contributors to predict how much work something will be and equally hard for a maintainer to then judge it for them^^
• Breaking changes will make all master branches of other repos that depend on it breaking. This means no other branch on those repos can merge and integrate it (and thus not be merged at all). This has two side-effects atm: It produces a ton of issues automatically and it stalls development.
• Some thought experiments (not yet encountered, but not unlikely especially as we get more and more repos): What do we do if we merge a pr that is breaking, only to then discover (probably much later) that it cannot be solved in upstream libs? Reverting changes is possible yet annoying and could mean a lot of work that will likely fall to maintainers and not the one who did the initial contribution. Same scenario but this time the contributor who introduced the breakage does not solve it on upstream libs. This means yet more work for maintainers. Both of these are extra work for maintainers, which is fine if the maintainer that has to deal with it has a chance to discuss that pr prior, but because we don't know if a pr breaks upstream libs that is hard to do. So this kind of overlaps with the first point.
• Overlapping with point 2: Third party libs working on snapshots will get constant (seemingly random) breakages because they always go to latest. And the other libs may or may not have fixed the breakage. This makes snapshots outside the internal arrow ecosystem unusable. Aside from that the snapshots don't work at all anyway atm.
There are probably things I am missing but those are the most pressing things atm I think
@simon.vergauwen and I were talking about a few things that we could possibly do do improve this. I'll post those in a thread for this message.
As always: Ideas/thoughts down below please ^^

[Jannis]

Two possible changes that could improve things:
• Fix every repo to a specific commit/snapshot build rather than always build against latest. Then when a dependency changes automatically produce a pr which ups this boundary to the latest snapshot (can also be done on a schedule if on commit is not possible). This means a commit in arrow-core cares none if it breaks anything in arrow-fx yet. arrow-fx will then get an automatic pr to "update" to a newer version, that may fail and need to be resolved. This has the benefit of the master branch in all repos ideally never breaking as it does now, which stalls development quite a bit. If this works the only downside of this is that a change in arrow-core is not seen as breaking before it is merged (but that already applies right now anyway). I think that this can be quite the elegant solution but it may be hard to coordinate and setup.
• Another idea is to be able to open pr's that point to different snapshot versions, that may or may not even come from a branch that is not master. This offers the ability to say open a breaking pr in arrow-core and have one condition for merging be that there are also green accepted prs in all affected repos that this change breaks. If it breaks fx we'd open a pr in fx that builds against core from the pr and not from master, and only if that pr is green and accepted we'd accept the pr in core. This is nice because a contributor directly sees where his code produces breaking changes. The downsides are: A single pr will built all of arrow again, coordination between prs is much more complex and the more arrow libs there are the crazier this will be.
The second one isn't great, I only included it for completeness and maybe this spawns some new ideas^^

[Jannis]

Another thing that may improve problem 4 is a special stable snapshot build that is updated on some frequency (nightly, on commit or whatever else may work)

[Rachel]

We could move the global check from push to pull request in order to know the consequences before

[Jannis]

that is a ~1h check every time though

[Rachel]

It could be done just for the dependent libraries

[Jannis]

It could be done just for the dependent libraries

That would be good for everything with few dependants. arrow-core will still have an hour long build. arrow-fx will also take around ~30-40 mins because of it and incubator taking both around ~15-20 mins to build. It is better don't get me wrong, but it still is quite the task if it is run on every commit for every pr

[simon.vergauwen]

but it still is quite the task if it is run on every commit for every pr

Well isn’t that the trade-off for knowing it beforehand without having to challenge yourself to anticipate all the breaks, no?

[simon.vergauwen]

Puts us back in our original basket with the split up code, no? 😄

We need to make 1.0 asap and then binary breakage is not an issue

[Jannis]

Well isn’t that the trade-off for knowing it beforehand without having to challenge yourself to anticipate all the breaks, no?

It certainly is. But I think that that is the least important problem atm. The constant breaking of master branches is way more important. Knowing what breaks beforehand would just be nice to have, but isn't essential

[raulraja]

We are working on making it stable. I think the trade-off is worth it and the same command can be added to Gradle if someone ever wants to check locally whether it's breaking other libs. We need to make arrow stable this year but it's gonna be painful.