Hello, We’re in the process of migrating the Grad...

# scripting

[bamboo]

Hello,

We’re in the process of migrating the Gradle Kotlin DSL from the legacy Kotlin script API to the latest experimental one so we can take advantage of the new implicit receivers feature but I’m having a little trouble figuring out how to migrate one of our script templates.

In the troublesome scenario the list of default imports must be computed dynamically from data passed from the outside to the compiler.

With the legacy API we were able to achieve it via a custom ScriptDependenciesResolver that reads the data from the given script resolver environment which can be configured via the -Xscript-resolver-environment command line argument. The custom resolver is setup via the ScriptTemplateDefinition annotation:

https://github.com/gradle/gradle/blob/6f683805c6b782b1eb91c01591aecabe6fe535fb/subprojects/kotlin-dsl/src/main/kotlin/org/gradle/kotlin/dsl/precompile/PrecompiledProjectScript.kt#L52-L54

My understanding is that it is NOT possible to mix the legacy ScriptTemplateDefinition annotation with the new KotlinScript one to get both, the dynamic implicit imports computed from the script resolver environment plus the implicit receivers from a custom compilationConfiguration setup.

So the question is then, does the experimental API allow for the list of default imports for a given script to be computed based on command line arguments passed to the compiler?

@KotlinScript(
        compilationConfiguration = DynamicDefaultImportsConfiguration::class
        // other settings
    )
    open class ScriptWithDynamicDefaultImports

    object DynamicDefaultImportsConfiguration : ScriptCompilationConfiguration({
        refineConfiguration {
            beforeCompiling {
                it.compilationConfiguration.with {
                    defaultImports(
                        // How to read arguments passed to the compiler from here?
                        TODO()
                    )
                }.asSuccess()
            }
        }
    })

[nastelaz]

cc @ilya.chernikov

[ilya.chernikov]

Hi @bamboo!
The previous solution is a bit hacky anyway, and we haven’t yet found time to analyze and maybe design a better way to access the compiler command line from scripts.
But, first of all, the old “environment” is still accessible internally via

get(ScriptCompilationConfiguration.hostConfiguration)?.get(ScriptingHostConfiguration.getEnvironment)?.invoke()

but more importantly, since you can use your own host, you can add a new key to the compilation configuration and pass required parameters via it directly, without using command line.
But maybe it’s a bit too early for gradle to switch to the new API anyway: I had an intention to implement a refinement callback for “sections”, to remove necessity to use the lexer directly on your side. But if you’re eager to use new API, I can probably try to prioritize this task as well.

[bamboo]

Hey @ilya.chernikov!

[bamboo]

Using the old environment should get things going for now, I’ll give it a try, thanks!

[bamboo]

since you can use your own host, you can add a new key to the compilation configuration and pass required parameters via it directly, without using command line.

[bamboo]

This is very interesting

[bamboo]

Do you have any pointers to the specific place in the docs or maybe an example that would make things a little more concrete? 🙂

[ilya.chernikov]

I fixed environment access code above a bit.

[bamboo]

I see 👍

[ilya.chernikov]

About examples:
Here is how you’re using a basic host implementation to compile your script - https://github.com/JetBrains/kotlin/blob/master/libraries/examples/scripting/jvm-simple-script/host/src/org/jetbrains/kotlin/script/examples/jvm/simple/host/host.kt#L18
And you can have a look at any definition of a configuration key, to see how to define your own. The configuration is a polymorphic container, which you can extend as you like just by defining new keys.

[bamboo]

Is it possible to use those same configuration keys to configure the Gradle Kotlin plugin compileKotlin tasks somehow?

[ilya.chernikov]

You mean in the case when you compile scripts with gradle?

[bamboo]

The scenario for these templates is a little different

[bamboo]

It’s not Gradle compiling the scripts directly

[bamboo]

but a regular compileKotlin task which has its configuration augmented by the kotlin-dsl plugin

[bamboo]

the kotlin-dsl plugin adds the required script templates to the classpath and configures the script environment via freeCompilerArgs

[ilya.chernikov]

As far as I remember you’ve been calling our compiler directly from gradle to compile build scripts. Is something changed here or we’re talking about different type of scripts?

[bamboo]

right, this is a different scenario

[bamboo]

this is to allow folks to implement Gradle plugins by writing Gradle scripts instead of classes

[bamboo]

so the templates are used for scripts in a regular Kotlin sourceset

[bamboo]

we call them precompiled script plugins

[bamboo]

or precompiled scripts

[bamboo]

we still have the scenario where Gradle compiles build script by calling the Kotlin compiler directly

[bamboo]

that scenario was mostly migrated to the new script API, except for the IDE support

[ilya.chernikov]

Ok, I see.

[ilya.chernikov]

First about precompiled scripts: if you want to use compileKotlin, then the compiler will use the script compilation configuration it can extract for the given script name. There are several ways to provide this configuration, but there is no direct functionality I can think of, that would allow you to pass data from kotlin-dsl extension to it.
So you can either continue to use “environment” hack, until we’ll come up with anything, or can create your own out-of-band data exchange.

[bamboo]

As long as the hack is working it shouldn’t be a problem. This is not directly exposed to users in any way.

[bamboo]

Meaning, we can evolve the internal protocol when we’re ready to.

[bamboo]

About the IDE support, the problem is a little similar

[bamboo]

We currently expose a ScriptDependenciesResolver to compute the classpath

[bamboo]

and we would like to tell the IDE the template has implicit receivers

[ilya.chernikov]

About the compilation from gradle, it will be nice if you can switch to the compilation via a host (or via more low-level script compiler & evaluator), as in the linked sample, rather than calling the compiler internals. And I will eventually make a replacement for the lexer usage, so you’ll be able to rely on the supported API instead (when it become really supported after graduating from experimental status, anyway).

[bamboo]

Yes, absolutely, we’re planning to switch to the new compilation method after 6.0 is released.

[ilya.chernikov]

And in all cases it will be nice if you’ll provide proper script templates with configurations attached with the KotlinScript annotation, In this case the IDE support should work out of box.

[bamboo]

We are switching to new template mechanism internally so we can replace the interface delegation we use in our templates by an implicit receiver instead

[bamboo]

In this case the IDE support should work out of box.

[bamboo]

It’s just not clear to me at this point what the replacement for the ScriptDependenciesResolver in that case would be

[ilya.chernikov]

I see the problem with script naming though - your precompiled scrips seems have the same file patter as the build scripts. We will have problems at the moment in distinguishing them.

[bamboo]

Maybe we should move the precompiled script templates to a separate module. For the batch compilation case we use the -script-templates compiler argument to tell exactly which templates should be used.

[ilya.chernikov]

It’s just not clear to me at this point what the replacement for the ScriptDependenciesResolver in that case would be

In your example above, the block:

refineConfiguration {
            beforeCompiling {

is one of the “resolvers” in the old API - it is a callback that is called by the compiler, so you can change compilation configuration according to your needs. So, e.g. now in this block you can add dependencies to the configuration.

[bamboo]

OR better, have a single set of templates and use them for precompiled scripts

[bamboo]

So, e.g. now in this block you can add dependencies to the configuration.

[bamboo]

I see, so we could make a blocking TAPI model call at that point

[ilya.chernikov]

Maybe we should move the precompiled script templates to a separate module.

The problem with distinction is IDE-specific, we can easily separate compilation, but in the IDE we have a single set of templates per project.

[bamboo]

there’s also ongoing work on the TAPI models

[bamboo]

The problem with distinction is IDE-specific, we can easily separate compilation, but in the IDE we have a single set of templates per project.

[bamboo]

Conceptually, the API exposed by the templates should be exactly the same

[bamboo]

and that’s why the IDE support currently work just fine with a single set of templates

[ilya.chernikov]

With dependencies?

[bamboo]

what do you mean?

[bamboo]

the dependencies are computed differently depending on where in the source tree we find the scripts

[bamboo]

(if it’s in a Kotlin sourceset we return the compileKotlin compile dependencies, otherwise we return the regular per script dependencies)

[bamboo]

the single resolver can handle it because the decision is taken by the TAPI model builder

[bamboo]

I hope that’s in the direction of your question 🙂

[ilya.chernikov]

More or less, except that in the IDE it is processed differently, the one in a sourceset are probably a part of a module, so the module dependencies are calculated via the project model. But we’re not decided what to do with scripts in this case. For a moment we ignore (in the IDE) all dependencies that are coming from the configuration for scripts and use module dependencies instead. But this blocks certain interesting use cases. So we may consider taking configuration dependencies into account.
For the out-of-sources scripts we’re using dependencies from configuration only.
So if you’ll have same base configuration for both script types, in the future you’ll need to decide which dependencies to add to the configuration.

[bamboo]

For a moment we ignore (in the IDE) all dependencies that are coming from the configuration for scripts

[bamboo]

Ah, I didn’t know that

[ilya.chernikov]

I mean only the scripts which are a part of a module/sourceset.

[bamboo]

Right. What about the default imports? Because I think we do compute them on the fly for different scripts and the IDE seems to honour them.

[bamboo]

I mean, different scripts which are part of a module sourceset.

[bamboo]

We give them different default imports to make certain extensions visible or not depending on the set of plugins they apply

[ilya.chernikov]

Everything else is fine, dependencies are not, because such “extending dependencies for a subset of the sources in the sourceset” seems not supported in the IDEA project model.

[bamboo]

I see 👍

[ilya.chernikov]

Coming back to the lexer stuff, what I’m planning to do is to have a callback like:

refineConfiguration {
            onSections("plugins", "dependencies") {

that would allow you to process sections of the scripts without using the lexer directly. What do you think about such an API?

[bamboo]

Currently, we compile the plugins / buildscript / initscript sections as separate scripts

[bamboo]

so we would basically need to be able to “cancel” the compilation of certain sections

[ilya.chernikov]

I don’t think you need a cancel, in this onSections block, you need to execute another compiler for the section before continuing with the results.

[bamboo]

also, I’m not sure we would be able to give up using the lexer completely because we take different decisions on how to compile scripts depending on which sections are present

[ilya.chernikov]

What in particular changing?

[bamboo]

to speed up repeated executions, we emit a class (that gets cached) that knows exactly which parts of the script need to be compiled

[bamboo]

this class is like a specialized interpreter for the script

[bamboo]

it makes loading scripts from the cache uniform

[bamboo]

and helps with keeping some things precompiled

[bamboo]

for a concrete example, take the following script:

plugins { java }

java { }

[bamboo]

upon analysing this script, we see that we always have to run the plugins block before we can compile the rest of the script

[bamboo]

so what we do is, we emit a Program (that’s what we call this uniform façade to the scripts)

[bamboo]

we compile the plugins section at this point because we know it needs to run every time

[bamboo]

and we make a Program that executes the precompiled plugins script and THEN compiles the rest after the plugins have been applied

[bamboo]

that strategy saves a few cycles when plugins or buildscript block changes cause the body of the script to be recompiled

[bamboo]

it is complicated though

[ilya.chernikov]

Ok, I see. Sounds complicated, yes.
So, seems this onSections callback doesn’t fit your use case. I’ll think about it further. I still don’t like the fact that you’re using compiler internals. Maybe the old (and still unused) solution with the source-sections compiler plugin should be revived after all.

[bamboo]

Hm, the compiler doesn’t seem to recognise the template configured via -script-templates anymore now that I changed from ScriptTemplateDefintion to KotlinScript

[bamboo]

(trying to test that environment access snippet)

[bamboo]

ok, it works if I use both annotations 🙂

[ilya.chernikov]

🙂 a nice hack

[bamboo]

for now it seems this will do

[bamboo]

I’ll play a bit more with it and let you know

[bamboo]

thanks again, Ilya

[ilya.chernikov]

Actually, you can use a complicated plugin option format:

-Pplugin:kotlin.scripting:script-definitions=...

or - better - use script definitions discovery mechanism.

[bamboo]

Status update: the migration to new KotlinScript annotation for compilation worked out great and Gradle 6.0 will ship with the new implicit receiver based script templates. Thanks again!

[bamboo]

We are still using the old style templates for the IDE support and now I’m looking into migrating those.

[bamboo]

Does IntelliJ already handle custom `compilationConfiguration`s?

[ilya.chernikov]

Thank you for the great news!

[ilya.chernikov]

Yes, IDE support should now be on the same or better level as with old templates.

[bamboo]

Great. I got IntelliJ to recognize my new @KotlinScript based template yesterday but I didn’t manage to get it to see the baseClass(...) I gave the template, will continue to investigate it later.

[bamboo]

One question though:

[bamboo]

is it possible to get the IDE to continue to use the resolver from @ScriptTemplateDefinition to get the classpath and default imports but get the remaining settings (mainly implicitReceivers) from the `@KotlinScript`’s compilationConfiguration?

[ilya.chernikov]

No, mixing is not possible, unfortunately. Why would you need it?

[bamboo]

As long as a refinement handler can make a blocking TAPI call to Gradle to get the data, there’s no need for it.

[bamboo]

Almost there! I got the IDE to recognize my template with an implicit receiver and the implicit receiver is taken into account for error checking BUT it is NOT taken into account for content assistance

[bamboo]

meaning, I don’t get suggestions for members of an implicit receiver type

[bamboo]

any thoughts?

[bamboo]

also it seems annotationsForSamWithReceivers has no effect in IntelliJ, should it be supported already?

[bamboo]

the experiment is here: https://github.com/gradle/gradle/pull/10977/files#

[bamboo]

maybe you can spot whether I’m doing something wrong

[bamboo]

ping @ilya.chernikov ☝️

[ilya.chernikov]

@bamboo, I see what is missing with annotationsForSamWithReceivers - will check whether I can fix it quickly.
About the implicit receiver - I need to dig a bit deeper.

[bamboo]

Thanks! Let me know if you’d like me create an YT issue for it.

[ilya.chernikov]

@bamboo, the problem with samWithReceiver is a bit worse then I thought. I need some more time to implement required functionality for the new scripting API. I’ll try to do it as soon as possible, but I’m by far not sure if it is not already too late for 1.3.60.
Anyway, here is the issue to watch - https://youtrack.jetbrains.com/issue/KT-34294

[bamboo]

Thanks, @ilya.chernikov

[bamboo]

My plan is to finish converting the remaining templates to uncover any remaining issues and give you a proper nightly you can use to validate the changes and the IntelliJ UX

[bamboo]

then we can see if the new templates go with Gradle 6.1 and Kotlin 1.3.60 OR if we have to wait for the next release train

[bamboo]

in parallel to this, there’s the new TAPI model work happening to optimize the script classpath computation

[bamboo]

I imagine there might be some overlap and the migration to the new templates might cause some rework on your side

[bamboo]

which is why I think getting a 6.0 compatible Gradle nightly with the new templates will be key

[bamboo]

how does that sound to you?

[ilya.chernikov]

Sounds good to me. Will try to push my part through as soon as possible.

[ilya.chernikov]

Hi @bamboo!
I think I fixed the problem with sam-with-receiver, although maybe the wrong one. 🙂
What I’ve meant with KT-34294 is that SAM with receiver is not properly processed then using the new API to compile it, rather than calling the internal compiler API directly, as you seems still doing. I fixed the processing, so if you will ever be able to switch to the new API - it should work.
But I also hopefully fixed the problem with IDEA, which wasn’t mentioned in the issue - I’m now configuring the sam-with-receiver annotation for IDEA as well. But I haven’t been able to test it yet - I have problems with trying to build your branch with locally-built kotlin.
The fixes will be in the next 1.3.60 EAP (in the next few days) so as soon as it is out and I have some time - I’ll try to test it again on your branch. Or you can try it yourself then.

[ilya.chernikov]

The problem with implicit receiver is being investigated by @nastelaz now.

[Tmpod]

@ilya.chernikov any update on implicit receivers recognition by IntelliJ? I'm trying to use them but I can't get the IDE to stop erroring on unknown references and to give me completion suggestions.

[ilya.chernikov]

@Tmpod the problem is fixed for upcoming 1.3.70 - https://youtrack.jetbrains.com/issue/KT-34740
(although unresolved references could be a sign of the different problem, so maybe it help if you'll give some more details, in a new thread.)

[Tmpod]

Great! The unresolved references are connected with the implicit receivers since I only get those on the methods the wrapping class has.

[Tmpod]

Is there any ETA for 1.3.70?

[ilya.chernikov]

~ Feb.

[Tmpod]

Cool

[Tmpod]

I'll just ignore those errors while 1.3.70 doesn't come out :p

[ilya.chernikov]

BTW, meanwhile you can try the EAP https://discuss.kotlinlang.org/t/kotlin-1-3-70-early-access-preview/15876

[Tmpod]

will do, thanks! totally missed this message too :/