Hey scripting enthusiasts! I have a new problem si...

# scripting

[Olaf Gottschalk]

Hey scripting enthusiasts! I have a new problem since I upgraded my product which includes the Kotlin script compiler to Kotlin 1.8.20... very annoying and I cannot really see what has changed besides the version number of Kotlin. It has worked as-is for several Kotlin versions and now breaks.

Inside my code I receive `String`s which contain code that needs to be compiled on the fly. The compile method takes the code snippet and calls eval on the script engine.

After the update to Kotlin 1.8.20 when my scripts contain String operations such as

val x = "Hello"
x.uppercase()

I get this horrible exception:
error: cannot access '<http://java.io|java.io>.Serializable' which is a supertype of 'kotlin.String'. Check your module classpath for missing or conflicting dependencies

The point is: I did not change anything there. I am using good old JVM 11 (due to some 3rd party libs that are very slow to upgrade, thanks go out to to SAP and IBM)

Looking at the Gradle dependency tree reveals nothing out of the ordinary, every Kotlin dependency is at 1.8.20. I do not see any conflicts there. And of course, my static code compiles just fine, it is just the dynamic code compiled at runtime that now complains.

Does anybody have any hints for me on how to check & proceed?
THANKS!

[ilya.chernikov]

Hi @Olaf Gottschalk
It looks like a bug to me. I've seen a couple of this sort recently, but I cannot figure out which might be your case.
Please file an YT issue, preferably with some additional info. The stacktrace may help, as well as some code or description how you're compiling/executing your script.
As a workaround, I can only suggest to rollback to 1.8.10 for now.

[Olaf Gottschalk]

The toughest thing here is that my script compiling function lives in a separate lib which is reused in many of our projects. All of them Gradle Projects, all of them targeting Java 11. In the project itself, a unit test compiling this script snippet works fine, but in an external project depending on our lib it breaks. This is so weird as I have been using Kotlin 1.6.x up to 1.8.10, always just bumping up the version number and everything worked, and now it doesn't anymore. How to set up a small project showcasing this is going to be very hard.
Do you have a tip on how to "see" what the compiler sees as the dependencies at the time of compilation?

[ilya.chernikov]

There is no need to setup a repro project, at least not now. I understand more or less what's going on - somewhere in the compilation path the java.home system properties is not processed properly. I just need to understand which path is missing the action. So if you'll show me the relevant part of the stacktrace and/or the code that set up script compilation, I might already be able to guess the reason and maybe offer a workaround.

[ilya.chernikov]

There is no way to see what script compiler uses as a compilation classpath, only debugging.

[ilya.chernikov]

If you want to try to fight it on your own, you can try also to add the JDK classpath to the regular scripting classpath manually.

[Olaf Gottschalk]

How would I denote the JDK classpath in terms of syntax? Is it just one lib name? Like kotlin-stdlib for the Kotlin lib?

[Olaf Gottschalk]

My biggest problem so far was that modifying the scripting classpath requires different settings depending on whether I run unit tests vs a production setup in docker vs fat jar vs a start within ImtelliJ. It's somewhat cumbersome to differentiate between all of these cases.

[ilya.chernikov]

I realized that you probably won't be able to get a modular (9+) SDK into classpath.
You'll need to pass the -jdk-home to the compiler instead.
But how to do it, depends on how you're launching scripts.

[Olaf Gottschalk]

I am not launching them. I use them to internally return a dynamic transformation function which is used to manipulate data.

[ilya.chernikov]

But what API are using for it?

[Olaf Gottschalk]

You mean JDK? Plain old 11.

[ilya.chernikov]

No, I mean scripting API. Are you using JSR-223 or BasicJvmScriptingHost or maybe something else?

[Olaf Gottschalk]

I'm using JSR-223

[ilya.chernikov]

I made a few more attempts to guess your configuration and reproduce something similar. I failed so far.
It could be much easier if you would provide me with some details, I mentioned earlier.
But I can give you another blind advice that might help.
If you're using plain default kotlin-scripting-jsr223 jar, you may try to set experimental kotlin.jsr223.experimental.resolve.dependencies.from.context.classloader system property to "true", before requesting an engine. It changes the way how the context dependencies are processed in the engine, and it may help.

[Olaf Gottschalk]

I will, @ilya.chernikov as soon as I am back at my computer - next week. 😔 Thanks again already! As it popped up with the version upgrade it must be related to some thing that changed in Kotlin 1.8.20. I know that's very vague! 🤣🙈

[ilya.chernikov]

I know what's changed, but I assumed I fixed all the affected code paths. And so far I cannot find the one triggered in your case.

[Olaf Gottschalk]

We'll find the path, I'll try to help as much as I can! If only I wasn't on holiday right now... 😉

[Olaf Gottschalk]

For all of you who might be interested in this issue, here's what @ilya.chernikov and I found as the culprit of my problems:

My project was started quite a long time ago, and since then, I was depending on an obsolete, old JSR223 engine factory that came with kotlin-script-util jar. In order to register this per config, I had a META-INF/services/javax.script.ScriptEngineFactory file which references this engine factory.

Up until Kotlin 1.8.10 this obsolete engine factory still worked properly, so I in fact never noticed (nor knew) anything about it.

Since Kotlin 1.8.20 it should NOT be used anymore. The new implementation is automatically registered when depending on kotlin-scripting-compiler-embeddable and kotlin-scripting-jsr233.

So, removing the obsolete dependency and removing the META-INF file solved my issue!

Thanks again, Ilya!

[ilya.chernikov]

In fact the usages of the kotlin-script-util should have been replaced with other APIs and jars quite long ago, but in 1.8.20 we accidentally broke it, and since it is long obsolete - have no intention to fix. We will remove it from publishing instead in one of the next versions.

[Christian Nedregaard]

@ilya.chernikov
We are also unable to upgrade from 1.8.10 -> 1.8.20 due to similar issues. Only fails in the fat-jar, not when running regular test.

Getting this error:

RuntimeException: Using JDK home inferred from java.home: /usr/lib/jvm/zulu11-ca-amd64
Using JVM IR backend
Using new faster version of JAR FS: it should make your build faster, but the new implementation is experimental
Loading modules: [<http://java.se|java.se>, com.azul.tooling, jdk.accessibility, jdk.attach, jdk.compiler, jdk.dynalink, jdk.httpserver, jdk.jartool, jdk.javadoc, jdk.jconsole, jdk.jdi, jdk.jfr, jdk.jshell, jdk.jsobject, jdk.management, jdk.management.jfr, jdk.naming.ldap, <http://jdk.net|jdk.net>, jdk.scripting.nashorn, jdk.sctp, jdk.security.auth, jdk.security.jgss, jdk.unsupported, jdk.unsupported.desktop, jdk.xml.dom, kotlin.stdlib, kotlin.script.runtime, kotlin.reflect, java.base, java.compiler, java.datatransfer, java.desktop, java.xml, java.instrument, java.logging, java.management, java.management.rmi, java.rmi, java.naming, java.net.http, java.prefs, java.scripting, java.security.jgss, java.security.sasl, java.sql, java.transaction.xa, java.sql.rowset, java.xml.crypto, jdk.internal.jvmstat, jdk.management.agent, jdk.jdwp.agent, jdk.internal.ed, jdk.internal.le, jdk.internal.opt]
Module kotlin.stdlib cannot be found in the module graph
Module kotlin.script.runtime cannot be found in the module graph
Module kotlin.reflect cannot be found in the module graph < CompletionException: java.lang.RuntimeException: Using JDK home inferred from java.home: /usr/lib/jvm/zulu11-ca-amd64

Using BasicJvmScriptingHost eval with this compilation configuration:

createJvmCompilationConfigurationFromTemplate<SimpleScriptTemplate> {
  jvm {
    // unpackJarCollections = true will enable this to work when running from the Spring Boot fat jar.
    dependenciesFromCurrentContext(wholeClasspath = true, unpackJarCollections = true)
  }
}

...and this evaluation config with a custom filtering classloader:

createJvmEvaluationConfigurationFromTemplate<SimpleScriptTemplate> {
  jvm {
    baseClassLoader(filteringClassLoader)
  }
}

I do not think kotlin-script-util is involved here?
Any idea how to proceed?

[ilya.chernikov]

The people has been reporting this for some time, e.g. https://youtrack.jetbrains.com/issue/KT-57907/Scripts-Module-kotlin.stdlib-cannot-be-found-in-the-module-graph-with-fatJar, but I don't have a good advice yet.
The reason is most likely the fat jar - it seems that the old ways of doing them do not handle the module-info files correctly, and that leads to the problems like this on the module-based JDKs (9+).

[Christian Nedregaard]

Thanks.

[Christian Nedregaard]

The workaround provided by Jon Peterson on the issue works like a charm for us. We are now successfully updated to Kotlin 1.9.0.

[István Mészáros]

Sorry for waking up the thread, but the mentioned workaround does not solve the problem completely for our case. The log output changes to this:

Loading modules: [myproject, java.base]
Loading modules: [java.base]
Unresolved reference: sql
Unresolved reference: Timestamp
Unresolved reference: Timestamp
Cannot access class 'java.sql.Timestamp'. Check your module classpath for missing or conflicting dependencies

And then lots of lots of similar messages are coming. I guess the problem is that only "myproject" and "java.base" is loaded and "java.sql" for example is missing.

Does anyone have an idea to fix this? Project works perfectly when running from IDEA or when it is deployed as separate JAR files.