<@UKPEELJCW> Would it be possible to add the abili...

# compiler

[Quantum64]

@dmitriy.novozhilov Would it be possible to add the ability to transform function parameter expressions and function call expressions to FirExpressionResolutionExtension?

[dmitriy.novozhilov]

I don't think so
I wonder how many troubles it may bring into IDE support

[Youssef Shoaib [MOD]]

Well, wouldn't addNewImplicitReceivers also cause issues in IDE? Or is arbitrary transformation more dangerous I guess?

[dmitriy.novozhilov]

addNewImplicitReceivers don't actually change FIR tree, so it looks quite safer
In IDE we need to guarantee the strict mapping between FIR and PSI nodes, and arbitrary transformations from plugins may easily break it

[dmitriy.novozhilov]

BTW I can not guarantee that FirExpressionResolutionExtension will exists in future at all
Originally it was added as an experiment, but we delayed actual design and prototyping any plugins with it because of focusing on K2 release

[Quantum64]

I am confused about how it would change IDE support. Doesn't the analysis API look at the fully resolved FIR tree?

[dmitriy.novozhilov]

Imagine that you make this transformation:

foo(a, b)
// ---->
foo(
    run {
        someCallback()
        bar(a)
    },
    b
)

And there is a problem with someCallback() call
Where it should be reported in IDE?
Or even if there are no problems, how to tell the user what exactly is happening here?

[Quantum64]

Ah, I see. I think it would be OK to link back to the PSI element that corresponded to the expression before it was transformed. Plugins would be responsible for making sure diagnostics generated by their transformed expressions make sense.

[Youssef Shoaib [MOD]]

This is a bit of a stretch, but maybe you could show a little highlight on a that, when you press alt-enter, has the option of showing you what the code is transformed into, and there you can see any errors in generated code.

[Quantum64]

I mean, do you not have this issue already with the variable assignment transformer?

[dmitriy.novozhilov]

There are also some implementation problems. E.g. when to allow plugins to transform arguments: before resolution of them or after?
First option is more or less useless, because there is no resolve information yet
Second option brings to necessity to rebuild part of control flow graph and changing smartcasts on it (actually, rolling them back and calculating new smartcast from scratch). And those smartcasts may change resolve of outer call (or transformed arguments may lead to choosing other overload)

[Quantum64]

So overload selection happens at some point after the arguments are resolved right? Can we not run transformers before that? (But after the arguments are resolved)

[dmitriy.novozhilov]

I mean, do you not have this issue already with the variable assignment transformer?

Assignment transformer is extremely restricted. Well right now it's not that restricted, but we will change it to disallow changing arguments in any way

[dmitriy.novozhilov]

So overload selection happens at some point after the arguments are resolved right? Can we not run transformers before that? (But after the arguments are resolved)

Not always. Regular arguments are resolved before resolution of call itself. But lambdas and callable references are resolved after candidate for containing call is already chosen

[Youssef Shoaib [MOD]]

What about the OverloadResolutuonByLambdaReturnType resolver? That one gets to decide between a few candidates and pick the best one. Could there not be an extension version of that that allows injection of arbitrary candidates? I'm thinking that, at that very specific point of time, it could be fine to allow transforming the Candidates, and then it's the plugins responsibility to, if it's going to change things dramatically, ensure that resolution for a particular expression is re-ran, perhaps through some other hook provided in the compiler

[Quantum64]

I didn't think of that, but I see why it is the case. Perhaps disallow lambda transformations?

[Quantum64]

I think the potential use cases for this feature are very broad and it's worth serious consideration. Being able to "decorate" parameters in this way could enable auto-boxing of union types implemented with sealed interfaces, seamless HKT emulation (by wrapping expressions with smart casts), and other functionality that I am interested in

[dmitriy.novozhilov]

What about the OverloadResolutuonByLambdaReturnType resolver

This one is very restricted as well (and part of my colleagues thinks that we shouldn't implement this feature at all, because it goes against all our resolution rules)

[dmitriy.novozhilov]

in this way could enable auto-boxing of union types implemented with sealed interfaces

I think this specific usecase will be solved when we introduce real union-types in the language itself. This is one of most important features which will we work on after K2 release

[Quantum64]

I definitely understand the concerns, however, I think users installing compilers plugins should know what they are signing up for. In my opinion, a good approach is designing the language in a consistent way that will work well for all users (as you have), but also making the language as extensible as possible through compiler plugins for users who need extra control and are willing to risk inconsistencies.

[dmitriy.novozhilov]

As for implementation complexity, I still convinced that it will we extremely hard to support in DFA
Just look over the code which is responsible for DFA and CFG building (keeping in mind that graph and smartcasts are build at the same time as resolution happens)
• ControlFlowGraphBuilder
• FirDataFlowAnalyzer

[dmitriy.novozhilov]

I think users installing compilers plugins should know what they are signing up for

It is correct for small projects where all code is owned by 1-3 people
But there might be problems in big projects, which may live more than one specific person work on it. So it is discusable topic too
I understand your concerns, but during language design we should consider all users, problems and benefits for each (at least each popular) user (and project) type

[Quantum64]

Maybe there could be a distinction between "safe" and "unsafe" compiler plugins where "safe" compiler plugins are restricted to a smaller subset of the compiler API that covers most use cases. Projects wishing to use "unsafe" plugins would have to opt-in to enable the extended API via a compiler flag.

[dmitriy.novozhilov]

This is an interesting idea

[Youssef Shoaib [MOD]]

I agree with the "unsafe" compiler plugin idea. I think certain compiler plugins that massively change the language (e.g. by changing resolution or editing expression bodies) obviously won't be stable in the eyes of the compiler, but providing such an API and making it opt-in can offer an interesting opportunity to experiment with ideas and to even internally use specialised compiler plugins that otherwise would be seen as too intrusive. I think a user opting in via a compiler flag and understanding the risks of doing so will also have the know-how to at least report bugs to the plugin developer and find ways of avoiding such bugs, and obviously it'll all be at the user's risk.

[raulraja]

I'm currently changing expression bodies in our arrow-meta prototype https://github.com/arrow-kt/arrow-reflection/blob/main/arrow-reflect-annotations/src/main/kotlin/arrow/meta/samples/Increment.kt#L41
We take the approach of compiling the template with an entire compiler pipeline that uses the current FirSession and adds the current declarations as tower scopes in the new generated file. Then we move the declaration over to the FirTransformer return override.
The template compiler triggers resolution on its own and runs the checkers again, this is the major drawback, but I'm controlling that with mutability to see if the current compilation comes from the regular compilation or template phase and only apply the transformation once per file.

I run this transformations as first step hacking a declaration checker for now and running just once over the FirFile .
https://github.com/arrow-kt/arrow-reflection/blob/main/arrow-reflect-compiler-plug[…]piler/plugin/fir/checkers/FirMetaAdditionalCheckersExtension.kt
Seems to work for all cases excepts resolution members added to new declarations.
for that we have to do it through the DeclarationGenerationExtension https://github.com/arrow-kt/arrow-reflection/blob/main/arrow-reflect-compiler-plug[…]/reflect/compiler/plugin/fir/codegen/FirMetaCodegenExtension.kt

We could also use a proper transformation extension point instead of the current declaration checker hack.
Would be ideal to have the checker context and diagnostic reporter in scope too. They are useful to add the current declarations in scope and to emit diagnostics as you perform the transformation over problematic nodes that can't be transformed.

[Quantum64]

Wow, that is extremely cursed. Certainly, the introduction of supported APIs would be preferable to a library as popular as Arrow shipping a hack such as this. 🙂

[raulraja]

I think even without hacks people are going to run in all kinds of issues because how the public api looks now.
We have FirElement.transform and FirElement.transformChildren which can be invoked as members of the class everywhere.
The trick is that there is only a couple of places in the entire frontend pipeline where using these methods make sense for compiler plugins that transform trees:
FirStatusTransformerExtension : too early for most elements
`FirDeclarationGenerationExtension`: probably right place but only gets called when you give it names to generate members
FirAdditionalCheckersExtension : too late.

[Youssef Shoaib [MOD]]

@raulraja there is also another place: in FirExpressionResolutionExtension.addNewImplicitReceivers you get access to every `FirFunctionCall`after it is resolved, so theoretically you can transform it and its children (and in fact I have done so initially).

Another place that isn't quite exposed, but just works, is `FirSessionComponent`s. You can replace any pre-existing one by calling FirSession.register() IIRC, and from that you get access to goodies like changing how conflict resolution works.

I think making these things explicitly allowed but also explicitly opt-in would be great, perhaps through having "unsafe" compiler plugins as a distinction.

[raulraja]

Thanks @Youssef Shoaib [MOD] for the hint. I tried this one in the past not for transformations but to actually add implicit receivers automatically that later I have to fix in IR.
The issue with that extension is that it just accepts calls but a compiler plugin user may want to modify other elements in the body and then once you use this interface FirExpressionResolutionExtension you need to fix whatever you add in backend IR.
This very example is something that is very frustrating. Most compiler plugin authors would want to perform a transformation and be done with it. The reality is that we have to go over multiple phases interfaces and implementations with different unrelated models to get it done FirCall, IrCall, bunch of utility classes etc. All these types have apis that are daunting and spread in different compiler utility classes. In general is a bad experience for the maintainer and library authors trying to keep up with the compiler.
I wish for a future where compiler plugins can be implemented with one just method or interface and it does not require you understand all the compiler internals and the implications of every single phase. At least for plugins that just want to transform the tree, type check the transformation and be done with it.

[dmitriy.novozhilov]

BTW does anybody know examples of good API in real production compilers?
I struggled to find something with similar abilities
AFAIK scala and rust allows to operate only with unresolved tree

[raulraja]

I'm somewhat familiar with scala macros and metaprogramming and find the quotes abstraction covers a couple of important use cases when working on macros or compiler plugins.
https://docs.scala-lang.org/scala3/reference/metaprogramming/macros.html
User already know how to write Kotlin, and that should be the default to generate and capture kotlin expressions. Similarly in Scala we do:

inline def assert(inline expr: Boolean): Unit =
  ${ assertImpl('expr) } // the ' is a quote that captures the expr arg at compile time

def assertImpl(expr: Expr[Boolean])(using Quotes) = '{
  if !$expr then
    throw AssertionError(s"failed assertion: ${${ showExpr(expr) }}")
}

There any args typed to Boolean are captured and turned into Expr[Boolean]. This allows the compiler to invoke assertImpl with reflection and pass in the Expr[Boolean] tree at compile time for the macro to expand. This expression tree is already typed but the compiler will allow a transformation in scope. Same you have something like Expr[_] there is also Type[_]

What we are trying to implement in Meta is inspired by this model where compiler plugin authors write just a function/annotation with a single function that can do what we have to do today in FIR in different phases:
Generate, Transform, Report diagnostics.

Currently looks like https://github.com/arrow-kt/arrow-reflection/blob/23a03a79ef620c41ad193dba5b7e398d[…]ect-annotations/src/main/kotlin/arrow/meta/samples/Increment.kt but it could be further simplified if we had quotes and were not dealing directly with complexity of FIR and Diagnostic API's. In this example I'm trying to get rid of the boilerplate required to encode Diagnostics in the compiler by hiding all that in the Diagnostics parent class and working around the amount of boilerplate required to create kotlin expressions with the FIR builder with "${+expression} + 1".call

The biggest problem IMO in meta-programming in Kotlin right now is that people need to understand the compiler internal API's and trees and they have the entire compiler in the classpath when they work with it. Would be easier to work with a representation and it's own api even if it's programmatic and not based in quotes or templates.

From the Arrow side we are grateful for FIR because many of the features we have today based on KSP are suboptimal and require users to perform full compilations in the IDE before they get proper error highlighting. All these compiler plugins working in the IDE as you type is the killer feature for FIR and the user experience.