I asked about this a while back, but I honestly fo...

# arrow

[Youssef Shoaib [MOD]]

I asked about this a while back, but I honestly forgot where the conversation led to. Is there any update on an efficient Option encoding in Arrow using `value class`es? If there isn't anything proposed right now, I think I might've come up with one that instead of one None object it uses a None(Int) class that indicates the level of nesting of the Option (sort of similar to the Failure class used in Result).

If there hasn't been any successful experiments for a nested Option, I could make a PR in the next hour or so for one, if the arrow team sees that it is fit for the library.

[simon.vergauwen]

Hey @Youssef Shoaib [MOD],

I think we're open to optimise Option, but a downside is that it would break when.

value sealed class is on the Kotlin Roadmap, and I am hoping we can use that to optimise the library. I guess we could also try using sealed interface + @JvmInline value class.

When introducing complexity like this I think we should also do benchmarks to proof we're actually making a difference. Since these kind of changes are binary breaking, so we'd have to plan it for Arrow 2.0.0 as well.

[Youssef Shoaib [MOD]]

The issue with sealed interface is that, AFAIK, the value class would get boxed whenever it is used as one of its superinterfaces, so that is a no-go for performance.

value sealed class is absolutely the ideal solution here, and checking the issue, it seems as though it'll be in preview for 1.7.20, and released in 1.8, so in fact because any value class based approach for Option will break binary-compatability, so it will take a while before it gets into Arrow anyways.

If you would like, I could still make a PR with the optimized version and name the type something like Maybe and run benchmarks on that to see if it is more performant. I'm not sure about Arrow's policy about introducing new public APIs in minor releases, but if adding a new type isn't considered a breaking change, then it possibly could be added in the meantime with RequiresOptIn until 1.8 comes out.

[simon.vergauwen]

Some of the core maintainers have been discussing Arrow 2.0, and changes/improvements we'd like to bring into Arrow. I'd love to make this a more public discussions.

Right now based on what you shared I think we might be better of waiting for 1.8, and use value sealed class + a breaking change in Arrow 2.0. That way we can maybe even keep the Option name, and the API.

We can introduce new stuff already though, as long as it's not breaking. If it's something we consider a 2.0 "candidate" it might be interesting to add it already and annotate it with RequiresOptIn or something like @FlowPreview, but @ArrowPreview.

[simon.vergauwen]

I just wanted to give you a heads up that it might not make it into Arrow, or might have to be postponed to Arrow 2.0 depending on what your findings are but if you're interested in writing an implementation with benchmarks then please do and we can discuss it further based on the findings 👍