https://kotlinlang.org logo
Join Slack
Powered by
I am testing the next Kotlin version 2.3.20 right ...
# scripting
o
I am testing the next Kotlin version 2.3.20 right now, using its Beta2 version with my scripting DSL language... when bumping the Kotlin dependency to 
2.3.20-Beta2
and running my tests, I hit hard errors when the script compiler compiles my code - it does not work at all and fails with these kinds of error messages:
Copy code
ERROR Cannot access built-in declaration 'kotlin.Any'. Ensure that you have a dependency on the Kotlin standard library.    
ERROR Cannot access built-in declaration 'kotlin.String'. Ensure that you have a dependency on the Kotlin standard library.    
ERROR Cannot access class 'kotlin.Pair'. Check your module classpath for missing or conflicting dependencies.
ERROR Unresolved reference 'run'.
and many more of the same kind. Something is wrong with the standard library. Plus: my tests with deliberate compile errors show problems with the generated 
ScriptDiagnostic
s which do not contain the correct 
location
anymore, just null. Is there any quick tip for me on what might be going wrong? THANKS!
i
Hi Olaf! Sorry for the delays. There were some noticeable changes in the scripting, mostly around allowing the compilation using LightTree parser (dropping the need for 
-Xuse-fir-lt=false
). But I do not understand yet, how it might lead to the problems that you're seeing. Could you please, show me how you're compiling and running your custom scripts in your tests?
o
Hi Ilya! Thanks for picking this up! I just did a quick check: updated my dependency from 
2.3.10
to 
2.3.20-RX2
- and sadly: it does not work. I get the same errors as mentioned, e.g. 
Cannot access built-in declaration 'kotlin.Any'. Ensure that you have a dependency on the Kotlin standard library.
This is the code that I use to compile my script strings:
Copy code
fun evalTransformation(sourceCode: SourceCode): ResultWithDiagnostics<EvaluationResult> {
    val compilationConfiguration = createJvmCompilationConfigurationFromTemplate<TraLaScript>()
    return BasicJvmScriptingHost().eval(sourceCode, compilationConfiguration, null)
}
The only thing that I control here is my 
TraLaScript
annotation class:
Copy code
@KotlinScript(
    displayName = "TraLa Script",
    fileExtension = TRALA_FILE_EXTENSION,
    compilationConfiguration = TraLaScriptConfiguration::class
)
abstract class TraLaScript
So, looking into 
compilationConfiguration
I do some magic with the classpath that is restricted to only a few allowed classes:
Copy code
object TraLaScriptConfiguration : ScriptCompilationConfiguration(
    {
        defaultImports(traLaDefaultImports)

        ide {
            acceptedLocations(ScriptAcceptedLocation.Everywhere)
        }

        refineConfiguration {
            beforeCompiling {
                it.script.scanForbiddenConstructs(blackListRegex)?.let { found ->
                    makeFailureResult(
                        found.map { (location, what) ->
                            "Do not use '$what': forbidden in TraLa".asErrorDiagnostics(locationWithId = location)
                        })
                } ?: it.compilationConfiguration.asSuccess()
            }
        }

        jvm {
            jvmTarget("17")
            compilerOptions(
                "-jvm-default=no-compatibility",
                "-Xcontext-parameters",
                "-Xreturn-value-checker=full",
            )

            fun List<*>?.log(msg: String) = msg + this?.joinToString("\n", prefix = "\n")

            val classPath =
                scriptCompilationClasspathFromContextOrNull(*requiredClasspath)?.also {
                    logger.debug { it.log("Using restricted transformation script classpath: ") }
                } ?: scriptCompilationClasspathFromContextOrNull(wholeClasspath = true).also {
                    logger.error { it.log("Unable to restrict transformation scripting classpath. Using whole classpath: ") }
                }

            updateClasspath(classPath)
        }
    }
)
I found out that my restriction of the classpath has to do with the error. But I do not understand why this worked with 2.3.10 but not with 2.3.20...
The restricted classpath I use is ONLY consisting of these entries which refer to libraries I want my customers to be able to use, including my own:
Copy code
private val requiredClasspath = arrayOf(
    "transformation-dsl",
    "jackson-annotations",
    "kotlin-xml-builder",
)
When I comment out this restriction, it works again...
Ok. I have quickly added a patch... which takes the name of my 
restrictedClasspath
key names and adds 
KotlinJars.kotlinScriptStandardJars.map { it.name }
to it, et voila: works!
Copy code
val kotlinJars = KotlinJars.kotlinScriptStandardJars
            val keyNames = requiredClasspath + kotlinJars.map { it.name }

            val classPath =
                scriptCompilationClasspathFromContextOrNull(*keyNames)?.also {
                    logger.debug { it.log("Using restricted transformation script classpath: ") }
                } ?:
                scriptCompilationClasspathFromContextOrNull(wholeClasspath = true).also {
                    logger.error { it.log("Unable to restrict transformation scripting classpath. Using whole classpath: ") }
                }

            updateClasspath(classPath)
So, my question boils down to: WHY do I explicitly have to add the Kotlin Standard jars names to the keyNames in version 
2.3.20
? Why is that not needed in 
2.3.10
?
And, @ilya.chernikov,while I now fixed this issue... I found a new rather strange one. I rely on compilation errors to come with a proper 
location
... one of my very basic tests put an unknown identifier straight into the code and uses the `ScriptDiagnostic`s 
location
property to highlight the error visually... but. In 
2.3.20
the location remains null... 😞
This is my deliberately broken script:
Copy code
foobar
So, there is just this ONE token, completely unknown. When I compile this with 
2.3.10
the 
ScriptDiagnostic
shows a location of 
Location(start=Position(line=1, col=1, absolutePos=null), end=Position(line=1, col=7, absolutePos=null))
- the same compilation with 
2.3.20-RC2
has a location of simply 
null
!
You see that the current version correctly points at line 1, columns 1 through 7. Exactly the word "foobar". But the newer version does not have this at all.
i
So, my question boils down to: WHY do I explicitly have to add the Kotlin Standard jars names to the keyNames in version 2.3.20? Why is that not needed in 2.3.10?
Oh, I see. I need to investigate, why exactly it happened, but I can imagine. The classloader structure and classpath extraction scheme are changed indeed. I created an issue to investigate it. Thanks for pointing out!
About the location. First, could you try it with some recent dev version of Kotlin (2.4.0-dev)? And second - could you try it (independently) with explicit 
-Xuse-fir-lt=false
and 
-Xuse-fir-lt=true
?
o
Thanks for investigating! Will check now with 2.4 and the setting!
What's the correct version to use? 
2.4.0-dev
cannot be found...
Explicit 
-Xuse-fir-lt=false
and 
-Xuse-fir-lt=true
with 2.3.20-RC2 does not change anything regarding the error location. It stays 
null
.
i
What's the correct version to use? 2.4.0-dev cannot be found...
You'll need to add the maven repo "https://maven.pkg.jetbrains.space/kotlin/p/kotlin/dev", and then use e.g. "2.4.0-dev-5881".
o
I'm sorry, Ilya, could you explain how I can use that version using Gradle version catalog? I added this in my build.gradle.kts in secion 
repositories
like this:
Copy code
repositories {
       maven { url = uri("<https://maven.pkg.jetbrains.space/kotlin/p/kotlin/dev>") }
       mavenCentral()
}
But still, it cannot use version 2.4.0-dev-5881: Plugin [id: 'org.jetbrains.kotlin.jvm', version: '2.4.0-dev-5881'] was not found in any of the following sources:
@ilya.chernikov... wow. Kotlin 2.3.20 was released? The bug that removes the error location in scripting is out now. Can you please help me find the reason why I do only see 
null
in the field 
location
?
I created a reproducer and an issue: KT-85029 Kotlin Scripting ScriptDiagnostics lacks error location in Kotlin 2.3.20
i
Ok, thanks for the repro. I'll check it, along with the stdlib issue, and report back.
❤️ 1
o
Stdlib needs to be added now, otherwise it never has access to it. That is a definitive change from 2.3.10 and before
i
Hi @Olaf Gottschalk! I found and fixed the issue about location. Preparing the changes for 2.3.21, hope the fix will arrive to it in time. But I'm not able to reproduce the stdlib issue or even understand how it might happen. The normal behavior is that unless you explicitly specify 
-no-stdlib
in compiler options, the stdlib is picked up automatically, and compilation will fail if it is not found. And this is the behavior I observe in my tests too. My testbed is the 
libraries/scripting/jvm-simple-script
test project in the kotlin repo. The 
org.jetbrains.kotlin.script.examples.jvm.simple.host.evalFile
contains the code that calls
Copy code
dependenciesFromCurrentContext(
                "scripting-jvm-simple-script" /* script library jar name */
            )
which is basically the same as your use of 
requiredClasspath
. If I debug it, I see that after this part is executed, the configuration's 
dependencies
property contains only a single 
scripting-jvm-simple-script.jar
, which is expected. Kotlin stdlib is added in the background, much like with the regular kotlin files compilation. And test passes, of course. I also tried to modify this project to resemble your definition more precisely, and the result is the same. So, I'm out of ideas for now. If you can make some repro project, that would be very helpful.
o
@ilya.chernikov in my reproducer for the location issue, if you take Kotlin 2.3.20 and comment out the line 
updateClasspath(KotlinJars.kotlinScriptStandardJars)
, it will show the problem with the missing stdlib. In Kotlin 2.3.10 (and before), you never needed to explicitly add the Kotlin StdLib...
@ilya.chernikov thanks for the two issues and fixes! I really wonder why I was the only one to find this, I thought your test coverage at JetBrains must be massive... 😉 So, the next bugfix release 2.3.21 will fix my problems, right? Sad that I was not able to get my point across before the release of 2.3.20... 😞
i
We generally have good test coverage. But there are always some holes. This part was significantly reworked around 2.3.20 and 2.4. Yes, the fixes are merged in the 2.3.21 branch. We'll have RC somewhere next week, I believe. Will be nice if you'll test it, just in case.
👍 1
I just checked again, and RC is planned for the week after Easter now.
o
Hi @ilya.chernikov, I am currently checking Kotlin 2.3.21-RC and I have mixed results, mostly positive. So, the error location is back! That's the good part. But there still is a notable difference between Kotlin 2.3.10 and 2.3.21: the script name differs. I don't know why. But I do not want this to yet be another regression in Kotlin! Can you please investigate, Ilya!? Here's a concrete diff of my unit test. Note that 2.3.21-RC has a strange name for my script: 
$b45e718d
. Does that make any sense to you? Cheers!
i
Hi @Olaf Gottschalk. That's intended change. You need to pass the name you want, when creating script source.
👍 1
o
Yeah, I found this - never really used the name. And the default used to be "script" I guess... When setting it, I get the correct name! Thanks!
👍 1
35 Views
Open in Slack
PreviousNext
Powered by