Hi, We have a situation where we need to repeat e...

# arrow

[Leo Laudouard]

Hi,

We have a situation where we need to repeat effect with Schedule.forever, or Schedule.spaced.
We have a memory leak and observed that there were a lot of instances of
arrow.fx.Schedule$Decision
arrow.fx.typeclasses.Duration
arrow.fx.ScheduleKt$repeatOrElseEither$$inlined$run$lambda$1$1$2

A basic example can highlight it: (with arrow 0.10.5)

import <http://arrow.fx.IO|arrow.fx.IO>
import arrow.fx.IO.Companion.effect
import arrow.fx.Schedule
import arrow.fx.extensions.io.concurrent.concurrent
import arrow.fx.extensions.io.monad.monad
import arrow.fx.fix
import arrow.fx.repeat
import arrow.fx.typeclasses.milliseconds

fun main(args: Array<String>) {
    effect {
      println("Looped")
    }.repeat(IO.concurrent(), Schedule.spaced(IO.monad(), 10.milliseconds))
      .fix()
      .unsafeRunSync()
}

Are we missing something ?
Thanks!

[Patrick Louis]

Hey,
I'm far from an expert for the Scheduling in arrow but I happen to use it in a similar way as you want to:

fun <A> repeatPolicy() = Schedule.withMonad(IO.monad()) {
    identity<A>() zipLeft spaced(AppConfig.updateFrequency.seconds) and forever()
}

My Guess is that you're missing the identity<A>() zipLeft part which basically folds the results of your scheduled executions into the identity of the latest run, dropping all other results.

As shown in the attached picture, we don't face any memory leak with this usage.

[simon.vergauwen]

Hey,
It's correct that those three instances are created per repetition in this case, but they should be eligible for GC continuously like seen in the screenshot shared Patrick. This might be further optimisable, but compared to sleeping this typically doesn't cause a problem.

In Arrow Fx Coroutines this would be more lightweight, but we've not made any comparison benchmarks so far. It might be worth optimising further in that library.

PS: @Patrick Louis which tool are you using there?

cc\\ @Jannis

[simon.vergauwen]

A Schedule that accumulates state can be indeed ignored, and dropped with zipXX as Patrick mentioned.

[simon.vergauwen]

@Patrick Louis you should be able to drop that and forever since spaced is build on top of forever while increasing the delay: Duration property of the Decision using desc.copy(delay = d + spacedParam).

[Patrick Louis]

Hey @simon.vergauwen,

[Patrick Louis]

the tool I used for the picture is grafana

[Patrick Louis]

Also thanks for the hint on and forever , I wasn't sure on the behaviour of spaced on it's own when I implemented it.

[Leo Laudouard]

Hey again,
Thanks both of you for your answer ! 🙂

@simon.vergauwen I am using visualVM to have live memory usage and I do not think those instances are eligible for GC

I just tried to run this piece of code, following @Patrick Louis advices

fun main(args: Array<String>) {
    effect {
        println("Looped")
    }.repeat(IO.concurrent(), repeatPolicy())
        .fix()
        .unsafeRunSync()
}

fun <A> repeatPolicy() = Schedule.withMonad(IO.monad()) {
    identity<A>() zipLeft spaced(1.milliseconds)
}

And I have the same result:

[simon.vergauwen]

Hey,
May I ask how you're using VisualVM? I've tried to use it with IDEA in the passed but had a lot of issues doing so.
I tried IDEA Ultimate (which the trial has expired for, and as a 47 Degrees employee I'm not eligible for an OSS license anymore like I was before).
Similar issue with JProfiler, which I've used the trail for.

[simon.vergauwen]

Your used heap still seems extremely small compared to your max heap, so I'm not sure if I'd expect GC to kick in here yet.

[simon.vergauwen]

Does VisualVM give any indication if or when GC was triggered?

[Patrick Louis]

Hey,
Maybe you could run the application with artificially llimited mem-size to see if GC collects the stuff or not.
i.e.

-Xms200M -Xmx200M

[Leo Laudouard]

Hey,

@simon.vergauwen I just use visualVM desktop application which list local java process, I have not tried to use it with IDEA

From visualVM you can manually trigger a GC, which is what I have done just before the screenshot and the heap dump.

I have just done the same test with jvm memory limits (Xms and Xmx at 100m) as @Patrick Louis suggests, and at some point I get an OOM exception:

Looped
Looped
Looped
Looped

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ForkJoinPool-1-worker-5"
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI TCP Connection(idle)" Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space

Corresponding visualVM heap graph and heap dump:

[Leo Laudouard]

Hi again,
Did someone had time to look into it ?
Can I do something ?

[simon.vergauwen]

Hey @Leo Laudouard,
I haven't had time to look deeply into this.

Yes, it'd be great if you could look test this with Arrow Fx Coroutines on the 0.11.0-SNAPSHOT, since that's the encoding we'll move forward with in the following releases.
It's probably also easier to debug that code, and we can then easily back-port a fix 🙂

[Leo Laudouard]

Hey @simon.vergauwen, for info, we just upgraded our projects to arrow 0.11, and the memory usage is fine now ! (I had no time to do it earlier, sorry about it)

[simon.vergauwen]

Awesome! Thanks for the update. I hope you're enjoying Arrow 0.11.0!