Hi, I have some issues with executing scripts from...

# scripting

[Egor]

Hi, I have some issues with executing scripts from our kotlin code.
1. Script evaluating consumes a lot of memory, we evaluate around 100 scripts and and it consumes about 2G of memory. Before 1.3.50 we noticed that if we use engine #compile, after that #eval(bindings) and after that we clear the bindings the memory consumption has been reduced to 700M (Not ideal but something). After upgrading to 1.3.50 this work-around does not work anymore.

2. I thought that using ‘kotlin-main-kts’ might reduce memory consumption, but I can not get it working with bindings. It works with executing simple scripts but not with using bindings ( or external params through engine.put(,))

I’d appreciate any any help, guys.
Thanks in advance.

[ilya.chernikov]

I assume that you’re talking about JSR-223 interface, right?
I can imagine a few things that can go wrong here. But it will be easier to guess, if I see some code. Do you have a repro?

[Christian Nedregaard]

@ilya.chernikov We are seeing similar issues with the JSR-223 engine on 1.3.50 (and earlier). We need to host a bunch of scripts that will be compiled once and executed frequently in single threaded contexts (one for each compiled script).

It seems like eval leaks memory.
The following test will give OOM at some point. One workaround is to define a function in the script and call it with invokeFunction instead of using eval. That seems to not be leaking memory.

Also we see that loading a single script engine instance requires about 650M of memory which seems very excessive. Subsequent instances does not consume so much memory.

Any feedback on how we can alleviate these things would be greatly appreciated. Especially the memory usage. We do not have to use the JSR-223 API.

Untitled

[ilya.chernikov]

I need to investigate it a bit further, but creating an engine with the new bindings leads to the expensive compiler internals loading. And maybe there are related leaks too. But if you’ll reuse the same engine, the things should be more lean.
See also this issue - https://youtrack.jetbrains.com/issue/KT-32020 - quite likely it is not actually a leak here, but the fact that execution history is preserved in bindings, if you’re using JSR-223. But it looks not too severe in comparison with your case.
You may have better control over it, if you’ll use scripting API without JSR-223, as it is described in the KEEP-75. There are references to some examples there too, this is by far not the best documentation, but may be enough to let you start.

[Christian Nedregaard]

Thanks @ilya.chernikov.
Initial investigations show that using BasicJvmScriptingHost directly and writing the script to define a function that is retrieved with valueOrThrow and then just calling that function is very efficient.
Low memory usage, no leak, no noticeable performance overhead.
Seems like the JSR-223 implementation needs some love 😉

[Egor]

@ilya.chernikov first of all thank you for quick response.
I did not have time to investigate the issue with using jsr-223 engine. Went all in to using scripting host instead. As a result evaluating same 100 scripts consumed something about 80M instead of 2G in case of jsr.

[ilya.chernikov]

JSR-223 is in fact a REPL which leads to much more overhead and ceremony, so this is quite expected. But of course there could be some additional leaks and bugs on our side.
Anyway, I glad that scripting host based approach works for you both.

[DALDEI]

This is execlent information. Ive been using JSR-223 for a while becuase it was the only model I could get to work (back in kotlin V(n-x) )
I still haven't found an self-contained example of scripting host that works 'right now' -- a huge legacy of outdated posts, blogs, comments, emails and quickly moving source tree --
Any suggestion for a 'currently known to work ' and ideally 'likely to keep working for while' minimal stub for using the BasicJVMScriptingHost ?

[ilya.chernikov]

The examples here - https://github.com/JetBrains/kotlin/tree/master/libraries/examples/scripting - are the part of the kotlin repo, so they will be kept in sync with the API updates.