Hi all I encounter some issues when I m trying to integrate kotlinlang #scripting

Hi all, I encounter some issues when I’m trying to...

Lorin

01/29/2019, 8:28 AM

Hi all, I encounter some issues when I’m trying to integrate kotlin-scripting (v1.3.11) into my Apache Flink (v1.7.1) application. One of the runtime dependencies of kotlin-scripting is kotlin-compiler-1.3.11.jar, which conflicts with Flink due to log4j.

Copy code

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.log4j.Level.toLevel(Ljava/lang/String;Lorg/apache/log4j/Level;)Lorg/apache/log4j/Level;
	at org.apache.log4j.helpers.OptionConverter.toLevel(OptionConverter.java:198)
	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:753)
	at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
	at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
	at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
	at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:81)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
	at org.apache.flink.client.cli.CliFrontend.<clinit>(CliFrontend.java:99)

So I use kotlin-compiler-embeddable-1.3.11.jar as the runtime dependency instead. The exception above is gone, but the following error occurs when evaluating kotlin script:

Copy code

Failure(reports=[ScriptDiagnostic(message=com/intellij/openapi/util/Disposer, severity=ERROR, location=null, exception=java.lang.NoClassDefFoundError: com/intellij/openapi/util/Disposer)])

Does anyone have idea how to solve this issue? Is it possible to make kotlin-compiler-embeddable-1.3.11.jar as the default runtime dependency of kotlin-scripting? Thanks!

Nikky

01/29/2019, 11:28 AM

how do you replace the dependency ?

Nikky

01/29/2019, 11:29 AM

i would like to replicate this

ilya.chernikov

01/29/2019, 4:28 PM

You should use

kotlin-scripting-jvm-host -embeddable

instead of the regular one. But it is available starting from 1.3.20 only. See also https://stackoverflow.com/questions/54160082/kotlin-1-3-breaking-slf4j-log4j/54219794#54219794

Lorin

01/31/2019, 3:12 AM

Hi @ilya.chernikov，I try to use the v1.3.20 embeddable version. The whole dependencies are as follows:

Copy code

kotlin-compiler-embeddable-1.3.20.jar
kotlin-reflect-1.3.20.jar
kotlin-scripting-common-1.3.20.jar
kotlin-scripting-jvm-1.3.20.jar
kotlin-scripting-jvm-host-embeddable-1.3.20.jar
kotlin-script-runtime-1.3.20.jar
kotlin-script-util-1.3.20.jar
kotlin-stdlib-1.3.20.jar
kotlin-stdlib-jdk8-1.3.20.jar
kotlinx-coroutines-core-1.0.1.jar
flink-dist_2.12-1.7.1.jar
flink-python_2.12-1.7.1.jar
log4j-1.2.17.jar
slf4j-log4j12-1.7.15.jar

But the following error occurs when evaluating kotlin script (an empty one):

Copy code

Failure(reports=[ScriptDiagnostic(message=java.lang.ExceptionInInitializerError, severity=ERROR, sourcePath=null, location=null, exception=java.lang.ExceptionInInitializerError)])

Any thought? Thanks a lot!

Nikky

01/31/2019, 3:16 AM

do you have your project public anywhere ?

Nikky

01/31/2019, 3:17 AM

or can you create a minimal reproducer ?

Nikky

01/31/2019, 3:19 AM

also wondering if you could fix your version mismatch with shadowJar yourself.. doing package relocation

Lorin

01/31/2019, 3:26 AM

Hi @Nikky, I will create a minimal demo. Send you later! Thanks for your help.

Nikky

01/31/2019, 3:28 AM

i am also updating my rather big program and seeing if that broke anything, even though it saves not much 2 MB are nice

Nikky

01/31/2019, 3:35 AM

so far everything still works for me..

Lorin

01/31/2019, 3:37 AM

Another question: when the exception above (such as java.lang.ExceptionInInitializerError) occurs with kotlin-scripting, how to get the Stack Trace?

Nikky

01/31/2019, 4:24 AM

feel free to check this out: https://github.com/NikkyAI/MinimalScriptingExample i just updated it and also organized the code a lot.. took some of the helper functions from my larger project

Nikky

01/31/2019, 4:26 AM

i am not entirely sure how to access the stacktrace when the exception initializer error basically produces no stacktrace..

ilya.chernikov

01/31/2019, 7:58 AM

The

ScriptDiagnostic

class inside the

Failure

has an exception field, which sees set in the case above. This exception contains a stracktrace. If it doesn’t work, then the easiest way is to debug evaluation setting an exception breakpoint for

ExceptionInInitializerError

ilya.chernikov

01/31/2019, 7:59 AM

It would be nice to get a stacktrace, without it I have hard times so far imagining what went wrong.

Lorin

01/31/2019, 8:26 AM

Hi @ilya.chernikov , the stacktrace is as follows:

Copy code

java.lang.ExceptionInInitializerError
	at kotlin.script.experimental.jvmhost.impl.KJvmCompilerImpl.compile(KJvmCompilerImpl.kt:87)
	at kotlin.script.experimental.jvmhost.JvmScriptCompiler.invoke$suspendImpl(jvmScriptCompilation.kt:54)
	at kotlin.script.experimental.jvmhost.JvmScriptCompiler.invoke(jvmScriptCompilation.kt)
	at kotlin.script.experimental.host.BasicScriptingHost$eval$1.invokeSuspend(BasicScriptingHost.kt:46)
	at kotlin.script.experimental.host.BasicScriptingHost$eval$1.invoke(BasicScriptingHost.kt)
	at kotlin.script.experimental.host.BasicScriptingHost$runInCoroutineContext$1.invokeSuspend(BasicScriptingHost.kt:35)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:32)
	at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:236)
	at kotlinx.coroutines.EventLoopBase.processNextEvent(EventLoop.kt:123)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:69)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:45)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:35)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at kotlin.script.experimental.host.BasicScriptingHost.runInCoroutineContext(BasicScriptingHost.kt:35)
	at kotlin.script.experimental.host.BasicScriptingHost.eval(BasicScriptingHost.kt:45)
	at com.datiip.yoeva.core.util.Engine.eval(Scripting.kt:32)
	at com.datiip.yoeva.app.ApplicationKt.main(Application.kt:31)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
	at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
	at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
	at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
	at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
	at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
	at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
	at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
	at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
Caused by: java.lang.RuntimeException: loader=sun.misc.Launcher$AppClassLoader@6e0be858
	at org.jetbrains.kotlin.com.intellij.openapi.util.Disposer.<clinit>(Disposer.java:40)
	... 32 more
Caused by: java.lang.NoClassDefFoundError: gnu/trove/Equality
	at org.jetbrains.kotlin.com.intellij.openapi.util.Disposer.<clinit>(Disposer.java:37)
	... 32 more
Caused by: java.lang.ClassNotFoundException: gnu.trove.Equality
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 33 more

Lorin

01/31/2019, 8:31 AM

The implementation of my com.datiip.yoeva.core.util.Engine is:

Copy code

object Engine {

    fun eval(script: SourceCode, env: StreamExecutionEnvironment): ResultWithDiagnostics<EvaluationResult> {
        val compilationConfig = createJvmCompilationConfigurationFromTemplate<Script> {
            defaultImports("java.io.File")
            jvm {
                dependenciesFromCurrentContext(wholeClasspath = true)
            }
        }
        val evaluationConfig = ScriptEvaluationConfiguration {
            constructorArgs(env)
        }
        return BasicJvmScriptingHost().eval(script, compilationConfig, evaluationConfig)
    }

}

The script api is:

Copy code

import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment
import kotlin.script.experimental.annotations.KotlinScript

@KotlinScript(
    displayName = "Yoeva script",
    fileExtension = "yoeva.kts")
abstract class Script(val env: StreamExecutionEnvironment) {
}

Nikky

01/31/2019, 8:35 AM

the script definition itself cannot be a abstract class (yet?)

Nikky

01/31/2019, 8:35 AM

make it a normal or open class

Lorin

01/31/2019, 8:42 AM

@Nikky making it a normal/open class doesn't work. The same issue occurs.

Nikky

01/31/2019, 8:44 AM

i notice you have no configuration specified in your annotation, thats something i did.. i doubt its required.. but..

Nikky

01/31/2019, 8:45 AM

but it does look like flink interferes in the calls somewhat

ilya.chernikov

01/31/2019, 9:13 AM

Oh, I see:

Copy code

Caused by: java.lang.ClassNotFoundException: gnu.trove.Equality

You need to add gnu.trove to your host dependencies (not to script dependencies)

ilya.chernikov

01/31/2019, 9:14 AM

Maybe we need to fix dependencies quite urgently

ilya.chernikov

01/31/2019, 9:15 AM

@Nikky -

the script definition itself cannot be a abstract class (yet?)

afaik, this is not the case, abstract classes should be supported. Do you have any experience otherwise?

Nikky

01/31/2019, 9:19 AM

abstract functions in abstract classes, afaik thats the gain of making something abstract correct ?

ilya.chernikov

01/31/2019, 9:26 AM

Ah, I see what you’re refering to. Not necessarily. You cannot have an abstract methods in this class, because of the issue you encountered with them, but you can still have an abstract class without abstract methods.

ilya.chernikov

01/31/2019, 9:31 AM

@Lorin, as far as I see, the dependency to the trove4j is correctly stated in the kotlin-compiler-embeddable pom: https://search.maven.org/artifact/org.jetbrains.kotlin/kotlin-compiler-embeddable/1.3.20/jar (see the last

<dependency>

entry). So the question is why it is not pulled in your case? It should be in the list of dependencies.

Nikky

01/31/2019, 9:31 AM

@Lorin can we have your output of

gradlew dependencies

Nikky

01/31/2019, 9:33 AM

and maybe

gradlew dependencyInsight  --dependency trove4j

Nikky

01/31/2019, 9:35 AM

assuming kotlin has it right i am guessing flink changes the dependency to a version that is not binary compatible

Lorin

02/01/2019, 2:48 AM

The issue is solved. Since kotlin-scripting is widely used in my flink applications, I treat kotlin-scripting and its dependencies as flink core dependencies which should be placed in flink lib directory and not packaged into application uber jar. As @ilya.chernikov point out, I missed the trove4j dependency. So after I put trove4j-1.0.20181211.jar into flink lib directory, the issue is gone. Thanks guys! BTW, kotlin-scripting is really an amazing feature!

👍 1

73 Views

Open in Slack

Previous Next