I'm thinking a little about the semantics for IorR...

# arrow-contributors

[Sam Pengilly]

I'm thinking a little about the semantics for IorRaise and have something in mind to get some comments on.

I'm taking another look at the Ior APIs after a previous discussion trying to figure out what a good approach would be for accumulating non-fatal errors and working cleanly with Eithers from within an Ior context. At the time I put in a helper function or two for dealing with IorNel but couldn't quite figure out what the next steps should be. I started thinking a bit about the sort of API that would be nice to use when you have a list of Eithers inside the context of an Ior and if any of them fail they should accumulate but otherwise be recovered to a value (or dropped) without short-circuiting the Ior.

Something like this:

fun parse(num: Int) = either {
  ensure(num % 2 != 0) { IllegalArgumentException("$num is even") }
  num
}

val result = iorNel<Throwable, List<Int>> {
  (1..10)
    .map { parse(it) }
    .mapNotNull { recover({ it.toEitherNel().bind() }) { null } }
}

Inside an ior builder, should recovering a bound Either that's a left case result in the error being combined into the Ior, putting it into the Both case? It's not necessarily explicit, but I'm not sure what other ways there are express this intent. It feels reasonable but also maybe not?
For this functionality to work, the only thing that would need to be done is that IorRaise would need to override raise like so:

override fun raise(r: Error): Nothing {
  combine(r)
  raise.raise(r)
}

Without that, the only thing that will ever cause an error to trigger IorRaise::combine is binding an Ior.Both, from what I can tell nothing else will ever cause an error to accumulate or put the outer Ior into a Both state.
I don't think that this override of raise will cause undesired effects with other cases like binding an Ior.Left, that should still short circuit the outer Ior into a Left case, but will just combine the error briefly in the IorRaise state first.

Could there be another option where mapOrAccumulate has slightly different semantics depending on whether it's used in an IorRaise context or a Raise context? Even if that's the path to go down I think it would still require an override of raise as defined above to work properly.

Right now the only approach to this is to be very explicit and convert the Either into an Ior.Both with the desired error information and a "recovered" value. Maybe that's the intention and I'm barking up the wrong tree. Interested in getting some thoughts on this as it would be good to have an easy to use API that makes a lot of sense intuitively.

[Sam Pengilly]

An approach using fold to convert the Either to an Ior.Both is far more explicit but a lot more verbose, I wonder if that sort of fold operation could have a name and be exposed within this builder context?

fun parse(num: Int) = either {
  ensure(num % 2 != 0) { IllegalArgumentException("$num is even") }
  num
}

val result = iorNel<Throwable, List<Int>> {
  (1..10)
    .map { parse(it) }
    .mapNotNull { either -> 
      either
        .toEitherNel()
        .fold(
          ifLeft = { Ior.Both(it, null) }, 
          ifRight = { Ior.Right(it) }
        )
        .bind() 
    }
}

[Youssef Shoaib [MOD]]

IorRaise<E> conceptually is the combination of a normal Raise<E> and also what I call an Accumulate<E> . I think what you actually want here is to have access to an accumulate function that simply adds an error to the accumulated values and that's it.
One can be built simply with: fun <E> IorRaise<E>.accumulate(value: E) { Ior.Both(value, Unit).bind()
Btw, your solution of overriding raise wouldn't actually work. Currently, the `ior`builder combines the raised value with any accumulated values, so this would just combine the raised value twice.
A better solution with accumulate would look like:

val result = iorNel<Throwable, List<Int>> {
  (1..10)
    .map { parse(it) }
    .mapNotNull { it.getOrElse { accumulate(it); null } }
}

The key here is that we explicitly accumulate the error when tAccumulate`, but I'd have to think about that moree either fails. I think a version of this idea can be made with

[Sam Pengilly]

mmm, that was another area I was considering, essentially thinking whether the combine function could be made public to be used by the lambda to ior {} but exposing it as an accumulate function does seem a bit cleaner

[Youssef Shoaib [MOD]]

I did exactly that in arrow-context, a little prototype of what Arrow could look like after contexts are released

[Sam Pengilly]

ah, by having both Raise<Error> and Accumulate<Error> as part of the context? Yea, I'm very keen for context receivers to eventually arrive and make it a bit easier to prototype ideas and extend things in my own code to prove out some ideas.

[Sam Pengilly]

I did also just play around with the idea of creating an extension function inside IorRaise with the signature: fun <A> Either<Error, A>.getOrAccumulate(recover: (Error) -> A): A that does the logic of your accumulate function above but streamlines it a little onto the existing type.

[Sam Pengilly]

with accumulate() as a base, if the use case is mostly just filtering a list for the successful values and accumulating the unsuccessful ones, I could also see something like a fun <A> Iterable<Either<Error, A>>.takeOrAccumulateAll(): Iterable<A> but that's starting to enter the territory of something that might be better to define as a separate extension. It'd just need context receivers to make it exist only inside an IorRaise scope

[Sam Pengilly]

at the very least it seems worthwhile to have a simple accumulate(Error) function in place, so I'll get that set up with some tests and docs and raise a PR with it

[Youssef Shoaib [MOD]]

Arrow 2 has forEachAccumulating which I forgot about. I think it can come in handy here:

val result = run { 
  val list = ArrayList<Int>(10)
  val errors = merge { // merge is available now, but it's too general and I'd like a specialized `attempt` version to exist
    // with contexts the iterable will be a receiver
    forEachAccumulating(1..10) {
      list.add(parse(it).bind())
    }
    return@run Ior.Right(list)
  }
  Ior.Both(errors, list)
}

You can also write this with recover (but I prefer the merge version, which has a specialized version currently in a PR as attempt):

val result = run {
  val list = ArrayList<Int>(10)
  recover({
    forEachAccumulating(1..10) { list.add(parse(it).bind()) }
    Ior.Right(list)
  }) { errors ->
    Ior.Both(errors, list)
  }
}

This is because forEachAccumulating goes thru the whole list no matter what, and hence the list will include all the successful elements, and errors will include all the failures. mapOrAccumulate uses this under the hood (in arrow-2).

[Youssef Shoaib [MOD]]

Perhaps an API like this can eventually be added to Arrow:

public inline fun <Error, A, B> Accumulate<NonEmptyList<Error>>.mapAndAccumulate(
  iterable: Iterable<A>,
  @BuilderInference transform: RaiseAccumulate<Error>.(A) -> B
): List<B> = buildList(iterable.collectionSizeOrDefault(10)) {
  recover({
    forEachAccumulating(iterable.iterator()) { item ->
      add(transform(item))
    }
  }) {
    accumulate(it)
   }
}

Which would work seamlessly in the iorNel builder