Hi I'm trying to implement a gradle plugin that is...

# scripting

[Wyatt Kennedy]

Hi I'm trying to implement a gradle plugin that is a script host and I'm running into some issues implementing the basic example [here](https://kotlinlang.org/docs/custom-script-deps-tutorial.html#create-a-script-definition). The script host project uses the plugin ``kotlin-dsl`` to create the gradle plugin which applies an old version of the kotlin plugin apparently. When the script definition project uses plugin kotlin("jvm") version "1.9.0", I get errors like /kotlin-stdlib-common-1.9.0.jar!/META-INF/kotlin-stdlib-common.kotlin_module: Module was compiled with an incompatible version of Kotlin. The binary version of its metadata is 1.9.0, expected version is 1.5.1 when compiling the host. When I explicitly set the plugin version in the definition project to "1.5.10" and down the version of kotlinx-coroutines-core to something that plays nice, it does kind've load and I can apply my gradle plugin in a project, but then I get errors like Unable to construct script definition: Unable to load base class kotlin.script.experimental.api.KotlinType@1f76d9c0 when trying to run a task that runs a script. The script definition code is an exact copy of the kotlin scripting example for now while I debug this, except the @KotlinScript is named ShaderDef. The host does this in a gradle task currently:

val configuration = createJvmCompilationConfigurationFromTemplate<ShaderDef> {  }
val results = BasicJvmScriptingHost().eval(extension.file.get().toScriptSource(), configuration, null)
results.reports.forEach {
    if (it.severity > ScriptDiagnostic.Severity.DEBUG) {
        println(" : ${it.message}" + if (it.exception == null) "" else ": ${it.exception}")
    }
}

I'll keep tinkering, but I'm really lost here. Has anyone else tried to use a custom gradle plugin as a scripting host?

[Wyatt Kennedy]

A little more context to explain what I'm really trying to do. I'd like to create a gradle plugin where you specify a list of kts scripts and then it will evaluate those scripts to generate configs from a DSL I'm writing. I want to do this during the processResources step of compilation for another project

[Wyatt Kennedy]

my only guess is that:

jvm {
    dependenciesFromCurrentContext(wholeClasspath = true)
}

isn't sufficient when the plugin is applied as a gradle plugin for some reason?

[mbonnin]

Gradle forces its own version of kotlin-stdlib at runtime, see https://github.com/gradle/gradle/issues/16345

[mbonnin]

What version of Gradle are you using though? Embedded kotlin 1.5.10 seems pretty old

[mbonnin]

You can try upgrading Gradle, downgrading your scripting dependencies and/or relocating your plugin jar

[Wyatt Kennedy]

Just using the version of gradle that intellij defaults to when creating a kotlin gradle project, which looks to be gradle-7.3. I figured it was doing so for a reason. Was this incorrect behavior in intellij 2023.1.3 and is there any harm in using a newer gradle version?

[mbonnin]

You should definitely bump to 8.2. There's no harm in using latest versions

[mbonnin]

It'll have Kotlin 1.8 https://docs.gradle.org/current/userguide/compatibility.html

[Wyatt Kennedy]

Alright, I updated gradle to 8.2 and updated my script definition to 1.8.20 which got rid of all the compatibility errors, thank you sir 🙂. but I'm still getting the unable to construct script definition: Unable to load base class kotlin.script.experimental.api.KotlinType@1f776f8b when trying to apply my plugin to another project so I'm guessing those compat issues were unrelated to the root cause here. Is there something funky about plugin classpaths where they don't automatically include their dependencies like org.jetbrains.kotlin:kotlin-scripting-jvm or something?

[mbonnin]

another project

Is that other project also using 8.2?

[mbonnin]

Is there something funky about plugin classpaths

There so many funky things with Gradle classpaths I don't even know where to begin

[mbonnin]

Let's say you're writing a plugin "com.example". Depending how the consuming build is applying your plugin and the other plugins applied, you might see different dependencies. And in all cases, the kotlin Stdlib will be the one embedded in Gradle, not the one "com.example" depends on

[mbonnin]

My rule of thumbs is: in your consuming build, make an included build and put all your plugins as dependencies there:

// build-logic/build.gradle.kts

dependencies {
  implementation("com.example:com.example-gradle-plugin:$version")
  implementation("org.jetbrains.kotlin:kotlin-gradle-plugin:1.8.20")
  // etc...
}

[mbonnin]

Unable to load base class kotlin.script.experimental.api.KotlinType@1f776f8b

Is there anything more to this error ? (stacktrace or so?)

[mbonnin]

There should be something like NoSuchMethod or ClassNotFound or something like this

[Wyatt Kennedy]

One moment, not by default by I'll run gradle with --stacktrace

[mbonnin]

Run ./gradlew --stop before that too just in case

[Wyatt Kennedy]

It won't let me post the full stacktrace, but at the bottom of the caused by chain it has Caused by: java.lang.ClassNotFoundException: org.ksentiment.ShaderDef
at kotlin.script.experimental.jvm.JvmGetScriptingClass.invoke(jvmScriptingHostConfiguration.kt:117)

that class ShaderDef is my script defintion class annotated with @KotlinScript.

My plugin project does includueBuild("../shader-script-definition) and then appears to find the dependency fine and compile it.

[mbonnin]

Interesting

[Wyatt Kennedy]

It's included without a version number in the script plugin project like so _implementation_("org.ksentiment:shader-script-definition")

[mbonnin]

Looks like it's using composite builds and dependency substitution

[mbonnin]

So look like in the end you have 3 builds, right?
• plugin build, composite, calls includueBuild("../shader-script-definition)
• shader-script-definition
• consumer build
?

[mbonnin]

How does consumer build includes the plugin? Do you publish it somewhere or is it a composite build too?

[Wyatt Kennedy]

a few more, but yes. The root project looks like this:

include("client")
includeBuild("shader-script-definition")

pluginManagement {
    includebuild("shader-plugin")
}

client is the consumer and is normal included in the root project.

[Wyatt Kennedy]

the client only mentions the plugin id like so:

plugins {
    kotlin("plugin.serialization") version "1.6.10"
    kotlin("multiplatform")
    id("org.ksentiment.shader-plugin") version "1.0-SNAPSHOT"
    `maven-publish`
}

[mbonnin]

Looks like it's only one big repo, right?

[Wyatt Kennedy]

yup that's right

[mbonnin]

I think you don't need the "1.0-SNAPSHOT" version (but that's probably not the cause of the issue)

[mbonnin]

I would maybe expect something like this:

include("client")

pluginManagement {
    includeBuild("shader-script-definition")
    includebuild("shader-plugin")
}

[mbonnin]

Because if org.ksentiment.ShaderDef is part of shader-script-definition , I would expect it next to the plugin

[mbonnin]

But it's hard to tell, there are so many different configuration possible...

[mbonnin]

You don't have a reproducer online by any chance?

[Wyatt Kennedy]

Tried moving the shader-script-definition into the pluginManagement block, but no dice, it's the same behavior. I don't yet, but I think that's the next step. I'm going to cut out all my other code and make a MRE with the bare minimum to present the issue. Ty sir for your help so far, I'll post here when I have a repo up

[Wyatt Kennedy]

Whipped up the bare minimum MRE here https://github.com/FatalCatharsis/gradle-plugin-kotlin-script-test-mre.git

The plugin only adds a task to the project that uses it. The task just tries to load a script file that I specify in a plugin extension. Produces the same error. To run it just use ./gradlew clienttest-kts-script

[mbonnin]

Thanks!

[mbonnin]

Task clientbuildEnvironment

------------------------------------------------------------
Project ':client'
------------------------------------------------------------

classpath
+--- project :plugin
| \--- github.fatalcatharsis:script-definition -> project :script-definition
| +--- org.jetbrains.kotlinkotlin stdlib jdk81.8.20 (*)
| +--- org.jetbrains.kotlinkotlin scripting common1.8.20 (*)

[mbonnin]

So at list buildEnvironment finds it

[mbonnin]

It might be because it's a transitive included build

[mbonnin]

Might be worth posting on the Gradle community slack. With the nice reproducer someone might be able to see what's wrong

[mbonnin]

It gets more interesting... 🤔 If I do this in your plugin apply() method, it finds a class:

val clazz = Class.forName("github.fatalcatharsis.ScriptDefinition")
        println("clasName '${clazz}'")

[mbonnin]

It's not about transitively included build, if you reduce down to 2 builds (client + plugin), there's still the same problem

[mbonnin]

Dumping the classloaders, I get this:

this.javaClass.classLoader = {VisitableURLClassLoader$InstrumentingVisitableURLClassLoader@13732} "InstrumentingVisitableURLClassLoader(ClassLoaderScopeIdentifier.Id{coreAndPlugins:settings[:]:buildSrc[:]:root-project[:]:project-client(export)})"
ScriptCompilationConfiguration::class.java.classLoader = {VisitableURLClassLoader@13733} "VisitableURLClassLoader(ant-and-gradle-loader)"

[Wyatt Kennedy]

Sorry stepped out for food. That is interesting, ty for the tip about buildEnvironment, could've used this many times now -_-. Was gonna try moving the script definition out of it's own build and include() it in the plugin build, but it sounds like you already tried that? VERY interesting that you can find the class with Class.forName and yet it still complains about no class def found.

[mbonnin]

Yea, there is definitely classloader funk down there 🕺

[mbonnin]

I traced it down to this line where some context classloader is used to load ScriptDefinition
https://github.com/JetBrains/kotlin/blob/5424c54fae7b4836506ec711edc0135392b445d6/[…]kotlin/script/experimental/jvm/jvmScriptingHostConfiguration.kt

[mbonnin]

But ScriptDefinition is not in that specific classloader

[mbonnin]

Now I have absolutely no idea why all this classloader funk is happening

[mbonnin]

Might be worth publishing your plugin to a repo, maybe included builds have a classloader of their own or so

[Wyatt Kennedy]

I've got a nexus server at home, I'll see if I can fanagle it so it's not using included builds. Would that confirm it's an issue with the classpath setup by the kotlin custom scripting code or a gradle classpath problem? We're well outside my realm of understanding

[mbonnin]

We're well outside my realm of understanding

100% same here, sorry 😅

[mbonnin]

The Gradle classloaders always leave me confused

[mbonnin]

What make me say it's a classloader thing is that
• this test passes: https://github.com/FatalCatharsis/gradle-plugin-kotlin-script-test-mre/pull/1/files#diff-536b036cef051e4e440adf5ce[…]d28d2b80c4f65a7898aefb90bbc3R10
• but the same thing from Gradle fails

[mbonnin]

+ The fact that there are 2 different classloaders for some unknown reason

[mbonnin]

Ah wait, I know the reason (maybe...)

[mbonnin]

Gradle needs kotlin-scripting-jvm on the classpath for their own build.gradle.kts

[mbonnin]

So it loads it, like kotlin-stdlib before any other dependency has any chance to load

[mbonnin]

Which means your classes are now split:
• kotlin-scripting-jvm and ScriptCompilationConfiguration is in Gradle "initial" classloader (whatever that is)
• ScriptDefinition is in Gradle "dependencies" classloader (the classloader that loads the jars from buildEnvironment)

[mbonnin]

I'm like almost 85% sure the Gradle "initial" classloader doesn't see ScriptDefinition

[mbonnin]

Ask the question on the Gradle slack, I'm sure people will have more details there

[mbonnin]

As to a solution/workaround, I hold by my "relocation should work" hunch.

[Wyatt Kennedy]

Now this is interesting. Used the mavenLocal instead to remove all the includeBuild links, published the script-definition and plugin projects (and plugin marker), and I get an error on gradle reimport:

> Could not resolve all files for configuration ':client:classpath'.
   > Could not find github.fatalcatharsis:script-definition:.
     Required by:
         project :client > github.fatalcatharsis.test-plugin:github.fatalcatharsis.test-plugin.gradle.plugin:1.0-SNAPSHOT > github.fatalcatharsis:plugin:1.0-SNAPSHOT

it recognizes that script-definition is a dependency of the plugin applied to client, but doesn't know where to find it even though I have mavenLocal in both pluginManagement repositories and dependencyResolutionManagement repositories. Something about it being a transitive classpath dependency of client makes it fail resolution. This is definitely a question for the gradle fellas now. @mbonnin you've been incredibly helpful sir. If you have a tip jar, I'd like to buy you a beer :).

[mbonnin]

Actually this seems to work 🎉

[mbonnin]

At least it does something different...

[mbonnin]

There's an API in kotlin-scripting-jvm that allows to pass a class that's going to be used for the context classloader

[Wyatt Kennedy]

And that worked without any fanagling of the gradle classpath?

[mbonnin]

Yep, no fanagling of anything!

[mbonnin]

$ ./gradlew :client:test-kts-script 

> Task :client:test-kts-script
 : Using new faster version of JAR FS: it should make your build faster, but the new implementation is experimental
 : /Users/mbonnin/git/gradle-plugin-kotlin-script-test-mre/client/example.test.kts (No such file or directory): java.io.FileNotFoundException: /Users/mbonnin/git/gradle-plugin-kotlin-script-test-mre/client/example.test.kts (No such file or directory)

BUILD SUCCESSFUL in 1s

[Wyatt Kennedy]

yeah the issue I had, had to do with default maven publications. After I fixed it and reran the test, I got the same class not found issues.

After adding the change you made, everything runs fine and my script evaluates no problem. I'm much more confused than when we started lol, but I'm glad I can move forward. So was the example missing anything on the kotlin tutorial, or do I just have to specify these classes I need in the script context because of me using it in a gradle plugin for some reason?

[mbonnin]

do I just have to specify these classes I need in the script context because of me using it in a gradle plugin for some reason?

Yes. I'm not 100% certain of the reason but my mental model is along the lines of this:
by default, the Kotlin scripting host uses the classloader of ScriptCompilationConfiguration (which is something from kotlin-scripting-jvm-host-1.8.20.jar). But since kotlin-scripting-jvm-host-1.8.20.jar is loaded super early by Gradle (because it also needs it for build.gradle.kts files), then this classloader doesn't know about things added later during the build evaluation, like your own ScriptDefinition here

[mbonnin]

By forcing to use ScriptDefinition::class.java.classLoader , this classloader obviously knows about ScriptDefinition

[mbonnin]

If you haven't already read it, I recommend this, it's a good intro to classpaths

[Wyatt Kennedy]

Got it, that actually makes a lot more sense. You are the real goat, sir, and I think it's about time I actually try to understand gradle's classpath shenanigans. Thanks for your assistance!