Hi <@U4UGS5FC7> we've been using arrow + reactor i...

# arrow

[mitch]

Hi @raulraja we've been using arrow + reactor in our kotlin projects. Pretty excited to see arrow getting better and more accessible! Quick question around option deprecation. The services that we're having are super deep into Option. We use that primarily for avoiding null types because it doesn't play well with Reactor's translation of nulls to Mono.empty. i saw the recommendation of null types (which is off the table for us) or typealiasing either to a maybe type which may work but wouldn't that mean we have to retrofit many of the extensions on options. Is there any way we can undo the deprecation of option in some way or maybe even reintroducing it as a Maybe type in Arrow itself which is isomorphic to either? There are potentially a lot of pushback internally with the typealiasing approach. I'm more than happy to contribute to the library if need be, Arrow's option is an important piece for us.

[raulraja]

Hi @mitch, I completely understand your concerns. We are not gonna introduce Option back in Arrow unless we can have special inlining for it like the Result type has in Kotlin. That means that is inlined an almost zero cost abstraction like nullable types, we plan to do this also for Either, attempt to inline most of it. If you can solve the problem of specialized inlined wrappers like Option with comparable performance to nullable types we can bring it back.

Can we see an example on how this is a problem with Reactor to see if there is a better alternative?

[stojan]

It's the same issue as RxJava (and other reactive streams implementations)

https://github.com/arrow-kt/arrow-core/issues/114#issuecomment-641224865

They simply do NOT support null

[raulraja]

@stojan in the case of Rx what is the role of Maybe? Is this is a problem exclusively for these two reactive framework implementations ? or does it happen elsewhere?. Perhaps Option can come back in specific integrations for those modules or a version of it that makes sense for Rx and Reactor but not as part of arrow-core, at least until someone proofs it can be inlined and has similar performance to nullable types.

[stojan]

in Rx io.reactivex.Maybe is an Observable that can emit 0 or 1 values (or can error). (Observable emits 0-n values (or can error).

So the use case for Observable<Option<T>> is to make the distinction between: the cache is empty, and the cache is not empty but has 0 items.

[stojan]

this is not really framework specific but it's Java interoperability specific (those frameworks are Java based, and don't have the concept of nullable types). So internally they use null to mean uninitialized (improving performance)

[stojan]

My suggestion for keeping it is explained in the same issue here: https://github.com/arrow-kt/arrow-core/issues/114#issuecomment-645474081

[stojan]

and you already (kind of) agreed with me here: https://kotlinlang.slack.com/archives/C5UPMM0A0/p1592552602410000?thread_ts=1592410792.370200&amp;cid=C5UPMM0A0 (I explain additional reasoning in the thread) but that never made it past slack comments to Github (mea culpa).

[stojan]

to sum up:

there are cases (Java interop, for example RxJava and Reactor) where nullable types just don't work.

I suggest that we keep Option in a separate, optional module, where we clearly point out it's not recommended but there as a last resort (for Java interop reasons).

I am happy to contribute in any such migration.

I don't expect such module to have more than 1 release, since Option is based on years of FP experience and algebraic laws

[stojan]

We are not gonna introduce Option back in Arrow unless we can have special inlining for it like the Result type has in Kotlin. That means that is inlined an almost zero cost abstraction like nullable types

it's an admirable goal, but when talking to the BE people in my company, most services are IO bound, so this is not really an issue. In cases where it is, they would probably never use it to begin with

[raulraja]

yes I get the use case and rationale people put around being I/O bound but that is not the case for all use cases of Arrow.

I suggest that we keep Option in a separate, optional module, where we clearly point out it’s not recommended but there as a last resort (for Java interop reasons).

I am happy to contribute in any such migration.

[raulraja]

Let’s do that!

[raulraja]

I don’t expect such module to have more than 1 release, since Option is based on years of FP experience and algebraic laws

We’ll see about that xD but yeah I get what you are saying. If this is a problem only surfacing in the JVM then arrow-optional fits and we can also include there the interop DSL to Java’s Optional beside nullables.

[stojan]

I'll create a ticket tomorrow, and we can go from there

[raulraja]

This is also potentially a opportunity to look at the internal encoding of Option and see if we can preserve the public API but optimize it as we mentioned the compiler is doing with Result. It may not be hard and if that were the case it can remain in core

[raulraja]

thanks so much @stojan for taking the lead 🙏

[raulraja]

and for explaining the reasons a couple of times 🙏

[mitch]

@stojan @raulraja really appreciate it! Let me know what I can do to help out as well.

[mitch]

Sorry for the late reply @raulraja. Reactor, being java based - also suffers from the exact problem described, (and to my understanding it was by design similar to RxJava Maybe type https://github.com/reactor/reactor-core/issues/2065)

for instance the line below, would compile - and resulted in a Mono.empty which is not exactly what’s intended. Mono.empty means the source doesn’t emit anything and unless handled in runtime using .defaultWhenEmpty(T) yields an infinitely hanging mono.

val monoNull: Mono<Int?> = Mono.just(5).map { null } // never resolves

To circumvent this we use Mono<Option<T>> as the return type. The trick here is we need a concrete type to box T? to make reactor behave sanely..

val monoOption: Mono<Option<T>> = Mono.just(5).map { none() } // resolves
val monoIdNull: Mono<Id<Int?>> = Mono.just(Id(null)) // resolves
val monoEither: Mono<Either<Unit, Int>> = Mono.fromCallable { Unit.left() } // resolves

hence the reason why Arrow’s Option<T> (and Either<L, R>) is very integral to ensure safety in our services. True we do lose a tiny performance, but it’s a more than acceptable tradeoff to the behaviourial correctness that we get by employing Arrow’s data types.

Regarding inline Option / Either, e.g. similarly kotlin.std’s Result<T> i think it’s going to be very cool when we get there. I guess how that is designed currently will have to change tremendously, as inline classes have so much restrictions around it. i.e. it can’t extend anything, idk if that’s going to work with higher kinds right now (maybe that’ll change with meta evolving).

[mitch]

I would be very hopeful to have option still in core - because it by design is different to what null is.

[raulraja]

Thanks for the detailed explanation @mitch

[raulraja]

Option and nullable types are not necessarily different, so far what I heard is more a design decision this frameworks have made regarding null that impacts kotlin main use case to have null in the type hierarchy as A?.

[mitch]

Option and nullable types are not necessarily different, so far what I heard is more a design decision this frameworks have made regarding null that impacts kotlin main use case to have null in the type hierarchy as A?.

I see your point, essentially ? is our wrapper similarly to how -> is the wrapper in kleislis. That does mean we’ll be passing nulls around under the hood. Some java libraries (like we’ve seen in rxjava and reactor) are sensitive to that.

How do we plan to approach inlining Either at the moment? I may need to check that out - i’m still keen to have a go inlining Option similarly. If there’s a way to keep option in arrow core and removing the performance penalty, i’m all for it.

[raulraja]

As mentioned with @stojan we have a path forward that in the worst case it would fit your use case. Still if we can prove that given a value of Option<A> and one of A? we can’t go back and forth without loosing info then it’s fine to keep Option.

Additionally I see Id in those examples as a valid wrapper which is still in core. What does Option give you that Id or Either don’t give you in your use case?

[raulraja]

is it a matter of too big of a refactor or are you finding other value in option besides tricking those framework to not loop on null values? Trying to make the best design choices, sorry for the many questions 🙂

[mitch]

👍 all good - i’m a big fan of arrow and i’m really keen to make it better. Coming from scalaz and cats it’s by far more understandable for the newcomers in my team who never really exposed to FP concepts.

Option itself resonates very well with developers - arguably it was easier for me to explain the fundamental concepts with option rather than either. We do believe conceptually the distinction between null and Option is pretty similar - but we prefer option regardless because nulls are null-pointer-exception-y - with options we avoid those altogether.

[raulraja]

All in favor for a better Option if it makes sense or a separate module to keep these kind of edge cases happy. Reading threads about it seems like this all went down to follow the reactive streams spec 😞

[raulraja]

That spec has old outdated stuff IMO and Rx should not have followed it as proposed with that issue on null. I don’t understand how that benefits its community but I may be missing something.

[mitch]

yeah i mean one way to think of it is about the weird directions that these frameworks have taken. However another way to think about it is on how Arrow is providing these framework users utopia. It’s almost a consensus at least in our department in the company that, i quote

If your kotlin project uses webflux, use Arrow option. Avoid nulls.

there’s another comment just popped in recently in the github issue I quoted saying the exact same problem and also a similar solution using java optional.

In kotlin we have Arrow’s Option which is lightyears better than that of Java’s. To lose that will deal a huge blow to many teams..

[stojan]

Well, prohibiting null did allow to use null as uninitialised internally, leading to better performance (in RxJava2 compared to RxJava1 which allowed null )

[stojan]

as promised: https://github.com/arrow-kt/arrow-core/issues/238

[simon.vergauwen]

Well, prohibiting null did allow to use null as uninitialised internally, leading to better performance (in RxJava2 compared to RxJava1 which allowed null )

There are other solutions for this, albeit ugly and harder to maintain.

[Youssef Shoaib [MOD]]

@raulraja Weird question, but can't you just have an inline class for Optional with a method to check if it's none (basically kind of like how Result works in Kotlin)? Because then you'd have almost no overhead. Or are you maybe bound by the previous Optional API that had Some and None?

[raulraja]

@Youssef Shoaib [MOD] yes beside the comment regarding inlining with the compiler there are better ways to encode Either and Option internally which would be more performant in Kotlin. Just haven’t gotten there yet. If you would like to explore reencoding those for the 1.x series to make them more performant I’d be happy to help.

[mitch]

@raulraja i’m keen to explore that - still thinking how, because inline class cannot extend, so that means it can’t extend Kind<ForOption, A> .. unless we have a different way to tackle HKT in 1.x ?

[raulraja]

Hi @mitch that is great!, HKT should not be a concern since we are gonna be able to potentially get rid of them at some point so it should not be in the poc. I think the best way to test this is to bench Scala Either, Option and Kotlin nullables with our own and look at what the compiler or the sources for Result look like. If it requires compiler intervention we will add it to meta so when it's compiled it's inlined.

If you need help with a similar setup the arrow Fx lib includes some benchs with Scala libs.

[raulraja]

Also very much interested in this going forward, please ping us if you have any questions or we can help in any way. 🙏☺️

[mitch]

@raulraja I think with inline classes we can potentially get Option similarly compiled to how Result is in terms of resulting code. (Haven't tested this out, but pretty sure). We'd have to sacrifice all the lightweight Kind<ForOption, A> though, but I'll trust that you'll have a way forward to model HKT in arrow 1.x? Slightly orthogonal to this, how might we model Option[Option[A]] with nulls now? Or do we need to box it with either?

[mitch]

And is this the benchmark code you're referring to? I'm going to start looking at this soon! Pretty excited https://github.com/arrow-kt/arrow-fx/blob/32d3d45d530f4dfddbf7f114f3cbc39d801491a3/arrow-benchmarks-fx/src/jmh/kotlin/arrow/benchmarks/Map.kt

[raulraja]

Hi @mitch, yeah we will get rid of kind inheritance requirements. It already works without it for List in meta which is a type we don’t extend and we are looking on how to include it in the arrow code base

[raulraja]

There is no way to model Option<Option<A>> with nullable types since they collapse the functor hierarchy and remove the nesting. I believe @stojan is working in moving Option out and there we can probably try the inline encoding.
Yeah those are the benchmarks. We can use them as template to benchmark the same programs over Option legacy, nullable types and new inlined option.