Hey folks. I know that there are people how are in...

# compiler

[dmitriy.novozhilov]

Hey folks.
I know that there are people how are interested in new FIR compiler and want to discover how it works, so I want to share with you some information about FIR itself and kotlin project environment setup so you can inspect, debug and even modify FIR locally for your experiments. Actually, this is only way to interact with FIR right now, since compiler API for FIR is postponed (see thread above)

Details in 🧵

[dmitriy.novozhilov]

First of all, little description about what FIR is and how it works:

FIR (aka Frontend IR) is absolutely new frontend for Kotlin compiler, which will replace existing frontend (aka FE 1.0) which is based on PSI, descriptors and BindingContext. There are two main goals for FIR project:
1. Increase compiler performance
2. Write new clean architecture of frontend (because architecture of FE 1.0 is actully piece of crap)

To achive this we decided to get rid of all structures from FE 1.0 which are listed above (even from PSI in some way) and replace it with new data structure named FIR, which is semantic (not syntax) tree which represents user code.

FIR tree is mutable tree. It is built from results of parser (PSI or other parser results) and in this raw form it is very close to PSI (it contains only information which is directly written in code). But at this stage we alredy performe some desugarigns to simplify futher resolve (e.g. we replace all if statements with when or thransform all for loops to while loops). After raw FIR is built, we pass it to number of processors, which somehow resolves code and represents different stages of compiler pipeline. Those stages are applied to all files of module in bulk. Here there are:

- IMPORTS -- resolve all import directives (find corresponding package for each import)
- SUPER_TYPES -- resolve all supertypes of all classes (resolve here means finding classId (fqn) for each type)
- SEALED_CLASS_INHERITORS -- find inheritors of sealed classes
- TYPES -- resolve rest explicitly default types of declarations (receivers, value parameters, return types)
- STATUS -- resolve and infer visibility, modality and modifiers of all declarations
- CONTRACTS -- resolve contracts on all functions
- IMPLICIT_TYPES_BODY_RESOLVE -- infer return types for all functions and properties without explicit return type (which includes analysis of their bodies)
- BODY_RESOLVE -- analyse bodies of all other fucntions and properties

There is also a last CHECKERS stage, which takes FIR and reports different diagnostics (warnings and errors)

After all those stages are completed resolved FIR is passed to fir2ir component, which produces backend IR, which is used by backend to generate platform code


This is very short overview, feel free to ask additional questions about it.

[dmitriy.novozhilov]

How to setup kotlin project to experiment with FIR (or IR or FE 1.0, whatever):
1. Clone/fork Kotlin project and setup environment variables to different JDK how it is described in ReadMe (https://github.com/JetBrains/kotlin)
2. Run ./gradlew dist. This command will compile all modules
3. Open project in IDEA:
- open folder with kotlin as project
- RMB on ./build.gradle.kts and chose Import Gradle project (or smt similar)
- wait until gradle build and indexing is over
4. ???
5. PROFIT

Now you can inspect code and run compiler tests using IDEA.

[dmitriy.novozhilov]

Module structure:

All compiler modules are lays in ./compiler directory, and all FIR related modules are in ./compiler/fir. Some main modules:
- :compiler:fir:raw-fir (with psi2fir and light-tree2fir) contains service which builds raw FIR from PSI
- :compiler:fir:tree contains almost all nodes of FIR tree
- :compiler:fir:cones contains classes for FIR type system
- :compiler:fir:resolve contains main logic of resolution with all compiler phases
- :compiler:fir:checkers
- :compiler:fir:fir2ir
- :compiler:fir:entrypont. This is entripoint (haha) to entire FIR compiler. Take a look for FirAnalyzerFacade if you want to discover how deep the rabbit hole goes

In kotlin project we have a lot of tests which checks different things. Mainly used tests:
- FirDiagnosticTestGenerated from :compiler:fir:analysis-tests. Those tests takes some program as input, give it to frontend and render all diagnostics which were reported by frontend back to original program (test data for them lays in ./compiler/fir/analysis-tests/testData/resolve)
- FirBlackBoxCodegenTestGenerated from :compiler:fir:fir2ir. Those tests also takes some program as input. But if in previous tests input program maybe anything (including incorrect code) then test data for those tests must not contains compile errors and should have fun box(): String in it. Test takes this program, runs FIR, fir2ir and JVM IR backend and then runs box method from compiled code. If box returned "OK" string then test is passed.

You can run and debug any of those tests to debug or write you own tests. Just add myShinyTest.kt to corresponding testData directory and run ./gradlew generateTests or Generate All Tests run configuration in IDEA, and it will add testMyShinyTest to corresponding test runner.

[dmitriy.novozhilov]

Using FIR compiler for compiling external project

Any Kotlin/JVM project may be compiled using FIR (and will be compiled succesfully with high probability). To enable FIR compilation you can just add -Xuse-fir to compiler arguments. It's better to use at most fresh compiler as possible, so I recommend you to use compiler which you build by yourself from sources.
1. run ./gradlew publish. This command will publish all compiler jars to local maven repository at path kotlinProject/build/repo
2. Add mavenLocal repository pointing to this directory to your project
3. Set kotlin version to 1.5.255-SNAPSHOT
4. Add -Xuse-fir to freeCompilerArguments
5. Enjoy

You also can attach debugger to process of compiler if you want:
- run your gradle task with next flags: -Dorg.gradle.debug=true -Dkotlin.compiler.execution.strategy="in-process". E.g. ./gradlew -Dorg.gradle.debug=true -Dkotlin.compiler.execution.strategy="in-process" build
- create new run configuration in IDEA from template Remote debug with port 5005
- run this configuration (in debug mode) after running gradle task

Please note that FIR is not completed, so using -Xuse-fir is not recommended for production purposes

[mikehearn]

Thankyou Dmitriy, that is very interesting. I have a few questions.

1. Could you elaborate more on the prior architecture and what made it problematic? Specifically, where do the performance improvements come from? All the phases and structure you describe sound pretty conventional.
2. To what extent can things run in parallel with FIR. Is the FIR tree owned by a single thread at once (presumably yes), if so how is parallel compilation managed - is parallelism only possible at the [B]IR level?
3. Although there's not (yet) any public API for writing FIR compiler plugins, how stable are the internal interfaces likely to be? If we want to write changes to the compiler that alter or insert new FIR stages, will there be constant merge conflicts or are the basic internal APIs quite stable now?
4. Why is body resolved twice, once for implicit returns and once for everything else? What's so special about implicit return types?

[mikehearn]

Background: I'm interested in writing a plugin/altering the compiler to do stuff with string substitutions, but only when the sink of the string substitution is annotated in a certain way. It feels like this is a good fit for the FIR level but it would be a JVM-specific feature, so, maybe it's more appropriate at the back end? I'm not sure if """${foo}""" type strings have been de-sugared by then.

[dmitriy.novozhilov]

1. Main problems of performance in FE 1.0 were related to very bad code locality and memory locality. All descriptors were lazy and because of that compiler always jumps between different parts of code, which kills number of JIT optimization. Also all information about resolution was stored in one big maps of maps (binding context), so CPU cannot cache objects good enough
2. Some of phase I described needs to global resolution (e.g. to resolve supertypes of some class we need to resolve supertypes of those supertypes), and some are very local (to resolve type of value parameter you don't need to access anything else except some provider which can found declaration by it's name). And this local phases may be parallelized (because they don't need have any data contention between threads). Also checkers stage may be parallelized because checkers can not modify the tree, only read data from it
3. FIR internals are not stable at all. Base architecture is mostly stable, but we may introduce some big changes if we decide that it may improve performance
4. Each body is resolved only once. But some bodies are resolved on one phase and other on another. IMPLICIT_TYPE_BODY_RESOLVE stage is needed to infer all return types of all functions/properties, which makes BODY_RESOLVE stage local (see 2.). On BODY_RESOLVE stage when analyzing some function call in some body we don't care is body of this function already analysed or not. We just need types (receiver, parameters, return types). So bodies in this stage can be resolved in parallel (and BODY_RESOLVE stage takes ~60-70% of time of all phases)

[dmitriy.novozhilov]

Background: I'm interested in writing a plugin/altering the compiler to do stuff with string substitutions, but only when the sink of the string substitution is annotated in a certain way. It feels like this is a good fit for the FIR level but it would be a JVM-specific feature, so, maybe it's more appropriate at the back end? I'm not sure if """${foo}""" type strings have been de-sugared by then.

If your changes doesn't affect resolve (and string template will have String type after your changes) then it is definitely should be a backend IR plugin. In FIR API we want to give API for providing new declarations (without bodies) and changing existing ones (e.g. change visibility). And filling bodies of generated declarations will be handled on backend side

[mikehearn]

In this case I want to replace String with some other expression that returns a String. Is that still better done in backend?

[dmitriy.novozhilov]

Yes. You can just replace string expression with something yours

[mikehearn]

Backend it is then. Final question - does FIR vs BIR make any difference w.r.t. IDE capabilities or integrations?

[mikehearn]

Like, if I were to write something for a hypothetical FIR plugin API, would IntelliJ be able to use that for something useful?

[dmitriy.novozhilov]

Yes, we are planning to make such FIR API so compiler plugins will work in IDE out of box

[galex]

Hello @dmitriy.novozhilov , what's the status of FIR?
I'm currently struggling to find how to build and test a compiler plugin with FIR to check things and return errors to the IDE (to see red lines under)

[dmitriy.novozhilov]

K2 compiler (FIR) is released in alpha version in Kotlin 1.7.0
K2 Kotlin IDE plugin still is an active development stage and can be build only from sources. For that you need to clone intellij-community repo and run run configuration named IDEA Community (K2 Kotlin)

[galex]

@dmitriy.novozhilov Thanks!
Is https://github.com/demiurg906/kotlin-compiler-course-examples/tree/compiler-plugin-project-setup the relevant place to look for an example how to build a K2/FIR Compiler Plugin?

[dmitriy.novozhilov]

It's a good template of K2 project setup (just update kotlin version to smth more fresh, like 1.8.20-dev-649)
For examples of plugin implementation I suggest to check official jetbrains plugins

[galex]

@dmitriy.novozhilov Thanks!
A little question about the testing framework and those tests here -> https://github.com/demiurg906/kotlin-compiler-course-examples/tree/compiler-plugin-project-setup/testData/diagnostics
Why are there two files, simple.kt and simple.fir.kt? Is the latter what FIR will generate and that's the instructions for the IDE to show an error?

[dmitriy.novozhilov]

simple.kt contains original code for test, and framework will past warnings and errors directly in it (like highlighting in IDE, yes)
simple.fir.txt contains text dump of FIR tree after analysis

To add new tests just add new .kt files and run generateTests gradle task