kotlinlang #arrow

Still very new to Arrow. I have a pretty simple use case I'm trying to implement: I want to read a File to do some processing. I'm starting with a function from a third-party library that can read the file but throws an exception if the format is wrong. Rather than wrapping the library, I thought I would try doing a layered approach where I lift the corresponding function into the context that I need. I'd probably end up with IO<Either<Error, Data>> maybe with some Async later on but, I'm having trouble with just transforming my existing function to use Either (it's pretty easy to wrap it and return Either<Error, Data> but I figure stuff like this might have already been factored out into reusable functions. I've tried

Either.catch()

but it expects

(Throwable) -> L

where I have an sealed hierarchy which stores a reference to the underlying Exception, which I can't easily pass in if I only have a handle to Throwable. It's also expecting a suspend function which is throwing a spanner into the works. I've tried

Either.functor().lift()

but can't quite figure out how to make it work. Right now I have an

Either<Data,Data>

. I'm sure there's a way to combine the two functions and probably a third one I don't know about to get the result I want. Is there some sample code for a similar use case or am I taking the wrong approach?

Satyam Agarwal

08/05/2020, 2:19 PM

Hei. I was wondering if I can achieve something similar with

IO

like we have in

Validated

by using Semigroup with applicative, we can get a list of all the failures. I see we have

IOSemiGroup

which has combine method, but I don’t it will give me the desired effect right ? Because

IO

encapsulates

Throwable

, so it will be always

IO<A>

while

Throwable

will be first failure while combining. I have seen the same thing with

parTraverse

IO

. I got the desired effect by giving

myIOOp.attempt()

parTraverse

. Then I get

IO<List<Either<Throwable, A>>>

, so with this I can extract out all failures. But it becomes complicated quickly.

Validated applicative

is neat, and uses

tupledN

which is nice so I don’t have to resort to coverting to

List

first. So, is there a way ?

simon.vergauwen

08/06/2020, 10:09 AM

When using Arrow Fx Coroutines

ForkConnected { // <-- Launches new fiber on `suspsend` context and cancels when `suspend` was cancelled.
  sleep(20_000.milliseconds)
  getLastLocation() // <-- suspend
}

Jörg Winter

08/06/2020, 5:42 PM

Hi, will arrow fx coroutines be released with 0.11 and does this release depend on Kotlin 1.4 or will 0.11 be released for 1.3.x ?

Tristan Blakers

08/07/2020, 4:51 AM

the perennial List<Either<A,B>> question comes up again - promise I have searched the archives! Have successfully used traverse with NEL to get

Either<NonEmptyList<A>,List<B>>

But can't figure out the "arrow way" to partition List<Either<A,B>> but retain all data. e.g. into separate List<A> and List<B>, or alternatively a Tuple2<List<A>,List<B>>

genovich

08/09/2020, 6:29 AM

Hi, does Arrow have function like

(List<IO<T>>) -> IO<T>

which runs each IO in

List<IO<T>>

in parallel and waiting for the first result?

than_

08/11/2020, 11:50 AM

hi, do we have API for

Ref

usage with arrow-fx-coroutines or do I still need to manually wrap the IO in suspend?

Clarence Dimitri Charles

08/11/2020, 2:34 PM

Hi, what happen if one these option is None :

Option.applicative().tupled(coursOpt, eleveOpt).fix().map { tuple -> // doing stuff }.

Is there a possibility to know which one has failed ?

Satyam Agarwal

08/12/2020, 8:02 AM

But what I would like to bring to your attention with this comment is that suspend is not a replacement for IO but all monads and foldable (see SequenceBuilder in the std lib) and can support interleaved transformer binding at all layers of the transformers

By this Raul means that suspend is replacement for IO including other things right 😅 ? Also, I was wondering if there would be any migration guide for switching the whole application code from

IO

Either + suspend

? I am assuming IO would eventually get deprecated when

arrow-fx-coroutines

will get mature ?

Satyam Agarwal

08/12/2020, 8:03 AM

Seems straight forward, but there are methods like

tupledN

parTraverse

parSequence

bracketCase

guarantee

from

IO

which would need some replacement thats not built on

IO

Alexander Schell

08/13/2020, 10:20 AM

Moin Moin! 🙂 Sorry, but here is the next dumb question: Thanks to @Patrick Louis I now know how to traverse a SequenceK in parallel - parTraverse. Unfortunately I don't see a way to use a thread pool with a limited number of threads during the traverse. Also I'm not sure if parTraverse is the best way to go: https://arrow-kt.io/docs/0.10/apidocs/arrow-fx/arrow.fx.typeclasses/-concurrent/par-traverse.html Could please somebody point out what I'm missing here? Thanks! Best regards Alex

spand

08/14/2020, 9:25 AM

I am looking for a structure that is sort of a combination of

Either

and

Pair

ie. it can have either a left or right or both. Does arrow have this and even if not does it have a standard name?

kioba

08/14/2020, 11:01 AM

👋 hello I am trying to update the arrow dependencies to the latest stable version. I couldn’t find the 0.10.5 version of

arrow-integration-retrofit-adapter

https://mvnrepository.com/artifact/io.arrow-kt/arrow-integration-retrofit-adapter Is that a mistake or the versions can be different on the integrations? If it needs an update, I am happy to help

Attila Domokos

08/14/2020, 4:04 PM

@simon.vergauwen - great job with

arrow.fx.coroutines

coming to Arrow 0.11.0!!! I rewrote a small app to use

parMapN

from that library and I am impressed with how the code looks: • No

IO

in functions, all impure functions are marked with

suspend

• The only function that has the dependency on arrow.fx.coroutines is the executor function, where I invoke the database operations with

parMapN

Really impressive work! ❤️ 💯

❤️ 4

:arrow: 9

Attila Domokos

08/14/2020, 7:05 PM

On a related note, it seems

parMapN

and

parTupledN

supports only arity 3, but I need to invoke it with 4 arguments. 😞 Should this be extended to at least arity 5?

pakoito

08/14/2020, 8:58 PM

https://github.com/arrow-kt/arrow-fx/blob/32d3d45d530f4dfddbf7f114f3cbc39d801491a3/arrow-fx/src/main/kotlin/arrow/fx/typeclasses/Concurrent.kt#L464

HieiJ

08/14/2020, 9:44 PM

While trying to write the example in the thread above I noticed that all the currying code in

arrow-syntax

works only for non-suspend functions. Could it be useful to have them also for suspend ones? (speaking about this: https://github.com/arrow-kt/arrow-core/blob/master/arrow-syntax/src/main/kotlin/arrow/syntax/function/currying.kt)

Gopal S Akshintala

08/15/2020, 3:20 AM

My understanding is

parTraverse

operation is different from

traverse

operation. It's not just

traverse

operation in parallel. If I am not missing something, aren't their names kind of misleading?

CLOVIS

08/15/2020, 1:16 PM

In the example:

Validated.applicative<KnownError>().map(vId, vName, vAge, { (id, name, age) ->
  Person(id, name, age)
}).ev()

Would it be fair to say that Applicative is ‘just' a generalization of

flatMap

that allows to call it on multiple objects? Maybe something like

vId.flatMap { id ->
  vName.flatMap { name ->
    vAge.flatMap { age ->
      Person(id, name, age)
    }
  }
}

Or is that completely wrong?

kioba

08/16/2020, 1:20 PM

I am trying to remove the

IO

and

ObservableK

from my library module and standardize calls around

using

Concurrent<F>

. Arrow version

0.11.0-SNAPSHOT

usage of

fx

mentions that

fx: MonadFx<F>' is deprecated. Overrides deprecated member in ‘arrow.typeclasses.MonadError’. Fx will be moved to each datatype as a DSL constructor.

What would be the correct usage for fx in this case?

simon.vergauwen

08/17/2020, 11:21 AM

The next version of the Arrow Fx website was published this weekend 🎉 Some additional improvements such as an

IO

migration guide for Arrow Fx Coroutines will follow soon, but all feedback is welcome! :arrow: https://arrow-kt.io/docs/next/fx/

🙌 3

👌 9

😍 2

CLOVIS

08/17/2020, 3:12 PM

Would it be truthful to say

EqK

is a utility that aims to make it possible to use a normal Eq implementation, but on the content of an object rather than the object itself? Eg. If we have

Eq<Car>

, we can use

EqK

to use our

Eq<Car>

in a

List<Car>

? I think that's what the documentation means by ‘lifting', but I'm really not sure.

Jörg Winter

08/17/2020, 8:54 PM

https://blog.jetbrains.com/kotlin/2020/08/kotlin-1-4-released-with-a-focus-on-quality-and-performance/ 🎉

:kotlin: 8

Phani Mahesh

08/18/2020, 9:41 AM

I have an

Either<E,V>

. I want to run some extra checks on

and end up with a

Left

if they fail. Is there a utility function?

Phani Mahesh

08/18/2020, 10:34 AM

I'm working with a bunch of traditional functions - I can

catch

to get a

Either<Throwable, *>

from a function that may throw. I'm looking for the reverse. I want to throw if right. Should I just create an extension function?

genovich

08/18/2020, 3:13 PM

Can anyone explain me difference between direct invocation of suspend function and wrapping it into

effect()

inside ConcurrentSyntax?

Manuel Roblek

08/18/2020, 9:45 PM

Hi, is compiling with Kotlin 1.4 supported atm? Anyway, @optics annotated classes don't seem to be able to be processed by kapt due to an exception in kotlin-metadata. I'm getting the following error with 0.10.5 and with 0.11.0-SNAPSHOT:

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':common:kaptKotlin'.
<...>
Caused by: kotlin.KotlinNullPointerException
	at me.eugeniomarletti.kotlin.metadata.KotlinClassMetadata$data$2.invoke(KotlinMetadata.kt:55)
	at me.eugeniomarletti.kotlin.metadata.KotlinClassMetadata$data$2.invoke(KotlinMetadata.kt:53)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
	at me.eugeniomarletti.kotlin.metadata.KotlinClassMetadata.getData(KotlinMetadata.kt)
	at arrow.optics.OpticsProcessor.getClassType(OpticsProcessor.kt:130)
	at arrow.optics.OpticsProcessor.onProcess(OpticsProcessor.kt:39)
	at arrow.common.utils.AbstractProcessor.process(AbstractProcessor.kt:108)
	at org.jetbrains.kotlin.kapt3.base.incremental.IncrementalProcessor.process(incrementalProcessors.kt)
	at org.jetbrains.kotlin.kapt3.base.ProcessorWrapper.process(annotationProcessing.kt:161)
	at com.sun.tools.javac.processing.JavacProcessingEnvironment.callProcessor(JavacProcessingEnvironment.java:794)
	at com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:705)
	at com.sun.tools.javac.processing.JavacProcessingEnvironment.access$1800(JavacProcessingEnvironment.java:91)
	at com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1035)
	at com.sun.tools.javac.processing.JavacProcessingEnvironment.doProcessing(JavacProcessingEnvironment.java:1176)
	at com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1170)
	at com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1068)
	at org.jetbrains.kotlin.kapt3.base.AnnotationProcessingKt.doAnnotationProcessing(annotationProcessing.kt:78)
	... 27 more

👀 1

Fred Friis

08/19/2020, 4:32 AM

//given
private fun getUserRole(user: User): Either<IError, SearchUserResponse> {
}

//this generates an explosion of calls
users
  .asSequence()
  .map { getUserRole(it) }
  .sequence(Either.applicative())
  .map { it.fix() }
  .map { it.toSortedSet(comparator) }

//all we want is to call getUserRole one by one
//AND short circuit on the first error
//ie if there's 100 users and we get an error on
//the 2nd one, we don't want to call getUserRole 
//another 98 times

Leo Laudouard

08/20/2020, 3:30 PM

Hi, We have a situation where we need to repeat effect with Schedule.forever, or Schedule.spaced. We have a memory leak and observed that there were a lot of instances of

arrow.fx.Schedule$Decision

arrow.fx.typeclasses.Duration

arrow.fx.ScheduleKt$repeatOrElseEither$$inlined$run$lambda$1$1$2

A basic example can highlight it: (with arrow 0.10.5)

import <http://arrow.fx.IO|arrow.fx.IO>
import arrow.fx.IO.Companion.effect
import arrow.fx.Schedule
import arrow.fx.extensions.io.concurrent.concurrent
import arrow.fx.extensions.io.monad.monad
import arrow.fx.fix
import arrow.fx.repeat
import arrow.fx.typeclasses.milliseconds

fun main(args: Array<String>) {
    effect {
      println("Looped")
    }.repeat(IO.concurrent(), Schedule.spaced(IO.monad(), 10.milliseconds))
      .fix()
      .unsafeRunSync()
}

Are we missing something ? Thanks!

Title

Leo Laudouard

08/20/2020, 3:30 PM

Hi, We have a situation where we need to repeat effect with Schedule.forever, or Schedule.spaced. We have a memory leak and observed that there were a lot of instances of

arrow.fx.Schedule$Decision

arrow.fx.typeclasses.Duration

arrow.fx.ScheduleKt$repeatOrElseEither$$inlined$run$lambda$1$1$2

A basic example can highlight it: (with arrow 0.10.5)

import <http://arrow.fx.IO|arrow.fx.IO>
import arrow.fx.IO.Companion.effect
import arrow.fx.Schedule
import arrow.fx.extensions.io.concurrent.concurrent
import arrow.fx.extensions.io.monad.monad
import arrow.fx.fix
import arrow.fx.repeat
import arrow.fx.typeclasses.milliseconds

fun main(args: Array<String>) {
    effect {
      println("Looped")
    }.repeat(IO.concurrent(), Schedule.spaced(IO.monad(), 10.milliseconds))
      .fix()
      .unsafeRunSync()
}

Are we missing something ? Thanks!

Patrick Louis

08/21/2020, 5:22 AM

Hey, I'm far from an expert for the Scheduling in arrow but I happen to use it in a similar way as you want to:

fun <A> repeatPolicy() = Schedule.withMonad(IO.monad()) {
    identity<A>() zipLeft spaced(AppConfig.updateFrequency.seconds) and forever()
}

My Guess is that you're missing the

identity<A>() zipLeft

part which basically folds the results of your scheduled executions into the identity of the latest run, dropping all other results. As shown in the attached picture, we don't face any memory leak with this usage.

simon.vergauwen

08/21/2020, 8:02 AM

Hey, It's correct that those three instances are created per repetition in this case, but they should be eligible for GC continuously like seen in the screenshot shared Patrick. This might be further optimisable, but compared to sleeping this typically doesn't cause a problem. In Arrow Fx Coroutines this would be more lightweight, but we've not made any comparison benchmarks so far. It might be worth optimising further in that library. PS: @Patrick Louis which tool are you using there? cc\\ @Jannis

Schedule

that accumulates state can be indeed ignored, and dropped with

zipXX

as Patrick mentioned.

@Patrick Louis you should be able to drop that

and forever

since

spaced

is build on top of

forever

while increasing the

delay: Duration

property of the Decision using

desc.copy(delay = d + spacedParam)

Patrick Louis

08/21/2020, 8:14 AM

Hey @simon.vergauwen,

the tool I used for the picture is grafana

👌 1

Also thanks for the hint on

and forever

, I wasn't sure on the behaviour of spaced on it's own when I implemented it.

👍 1

Leo Laudouard

08/21/2020, 9:07 AM

Hey again, Thanks both of you for your answer ! 🙂 @simon.vergauwen I am using visualVM to have live memory usage and I do not think those instances are eligible for GC I just tried to run this piece of code, following @Patrick Louis advices

fun main(args: Array<String>) {
    effect {
        println("Looped")
    }.repeat(IO.concurrent(), repeatPolicy())
        .fix()
        .unsafeRunSync()
}

fun <A> repeatPolicy() = Schedule.withMonad(IO.monad()) {
    identity<A>() zipLeft spaced(1.milliseconds)
}

And I have the same result:

simon.vergauwen

08/21/2020, 11:48 AM

Hey, May I ask how you're using VisualVM? I've tried to use it with IDEA in the passed but had a lot of issues doing so. I tried IDEA Ultimate (which the trial has expired for, and as a 47 Degrees employee I'm not eligible for an OSS license anymore like I was before). Similar issue with JProfiler, which I've used the trail for.

Your used heap still seems extremely small compared to your max heap, so I'm not sure if I'd expect GC to kick in here yet.

Does VisualVM give any indication if or when GC was triggered?

Patrick Louis

08/21/2020, 12:06 PM

Hey, Maybe you could run the application with artificially llimited mem-size to see if GC collects the stuff or not. i.e.

-Xms200M -Xmx200M

Leo Laudouard

08/21/2020, 12:41 PM

Hey, @simon.vergauwen I just use visualVM desktop application which list local java process, I have not tried to use it with IDEA From visualVM you can manually trigger a GC, which is what I have done just before the screenshot and the heap dump. I have just done the same test with jvm memory limits (Xms and Xmx at 100m) as @Patrick Louis suggests, and at some point I get an OOM exception:

Looped
Looped
Looped
Looped

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ForkJoinPool-1-worker-5"
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI TCP Connection(idle)" Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space

Corresponding visualVM heap graph and heap dump:

Hi again, Did someone had time to look into it ? Can I do something ?

simon.vergauwen

08/25/2020, 7:51 AM

Hey @Leo Laudouard, I haven't had time to look deeply into this. Yes, it'd be great if you could look test this with Arrow Fx Coroutines on the

0.11.0-SNAPSHOT

, since that's the encoding we'll move forward with in the following releases. It's probably also easier to debug that code, and we can then easily back-port a fix 🙂

Leo Laudouard

12/18/2020, 2:03 PM

Hey @simon.vergauwen, for info, we just upgraded our projects to arrow 0.11, and the memory usage is fine now ! (I had no time to do it earlier, sorry about it)

simon.vergauwen

12/18/2020, 6:47 PM

Awesome! Thanks for the update. I hope you're enjoying Arrow 0.11.0!

View count: 3