https://kotlinlang.org logo
d

dmitriy.novozhilov

05/31/2021, 4:24 PM
Hey folks. I know that there are people how are interested in new FIR compiler and want to discover how it works, so I want to share with you some information about FIR itself and kotlin project environment setup so you can inspect, debug and even modify FIR locally for your experiments. Actually, this is only way to interact with FIR right now, since compiler API for FIR is postponed (see thread above) Details in 🧵
👍 28
👍🏾 1
💯 18
👍🏼 3
K 16
First of all, little description about what FIR is and how it works: FIR (aka Frontend IR) is absolutely new frontend for Kotlin compiler, which will replace existing frontend (aka FE 1.0) which is based on PSI, descriptors and
BindingContext
. There are two main goals for FIR project: 1. Increase compiler performance 2. Write new clean architecture of frontend (because architecture of FE 1.0 is actully piece of crap) To achive this we decided to get rid of all structures from FE 1.0 which are listed above (even from PSI in some way) and replace it with new data structure named FIR, which is semantic (not syntax) tree which represents user code. FIR tree is mutable tree. It is built from results of parser (PSI or other parser results) and in this raw form it is very close to PSI (it contains only information which is directly written in code). But at this stage we alredy performe some desugarigns to simplify futher resolve (e.g. we replace all
if
statements with
when
or thransform all
for
loops to
while
loops). After raw FIR is built, we pass it to number of processors, which somehow resolves code and represents different stages of compiler pipeline. Those stages are applied to all files of module in bulk. Here there are: -
IMPORTS
-- resolve all import directives (find corresponding package for each import) -
SUPER_TYPES
-- resolve all supertypes of all classes (resolve here means finding
classId
(fqn) for each type) -
SEALED_CLASS_INHERITORS
-- find inheritors of sealed classes -
TYPES
-- resolve rest explicitly default types of declarations (receivers, value parameters, return types) -
STATUS
-- resolve and infer visibility, modality and modifiers of all declarations -
CONTRACTS
-- resolve contracts on all functions -
IMPLICIT_TYPES_BODY_RESOLVE
-- infer return types for all functions and properties without explicit return type (which includes analysis of their bodies) -
BODY_RESOLVE
-- analyse bodies of all other fucntions and properties There is also a last CHECKERS stage, which takes FIR and reports different diagnostics (warnings and errors) After all those stages are completed resolved FIR is passed to fir2ir component, which produces backend IR, which is used by backend to generate platform code This is very short overview, feel free to ask additional questions about it.
How to setup kotlin project to experiment with FIR (or IR or FE 1.0, whatever): 1. Clone/fork Kotlin project and setup environment variables to different JDK how it is described in ReadMe (https://github.com/JetBrains/kotlin) 2. Run
./gradlew dist
. This command will compile all modules 3. Open project in IDEA: - open folder with kotlin as project - RMB on
./build.gradle.kts
and chose
Import Gradle project
(or smt similar) - wait until gradle build and indexing is over 4. ??? 5. PROFIT Now you can inspect code and run compiler tests using IDEA.
Module structure: All compiler modules are lays in
./compiler
directory, and all FIR related modules are in
./compiler/fir
. Some main modules: -
:compiler:fir:raw-fir
(with
psi2fir
and
light-tree2fir
) contains service which builds raw FIR from PSI -
:compiler:fir:tree
contains almost all nodes of FIR tree -
:compiler:fir:cones
contains classes for FIR type system -
:compiler:fir:resolve
contains main logic of resolution with all compiler phases -
:compiler:fir:checkers
-
:compiler:fir:fir2ir
-
:compiler:fir:entrypont
. This is entripoint (haha) to entire FIR compiler. Take a look for
FirAnalyzerFacade
if you want to discover how deep the rabbit hole goes In kotlin project we have a lot of tests which checks different things. Mainly used tests: -
FirDiagnosticTestGenerated
from
:compiler:fir:analysis-tests
. Those tests takes some program as input, give it to frontend and render all diagnostics which were reported by frontend back to original program (test data for them lays in
./compiler/fir/analysis-tests/testData/resolve
) -
FirBlackBoxCodegenTestGenerated
from
:compiler:fir:fir2ir
. Those tests also takes some program as input. But if in previous tests input program maybe anything (including incorrect code) then test data for those tests must not contains compile errors and should have
fun box(): String
in it. Test takes this program, runs FIR, fir2ir and JVM IR backend and then runs
box
method from compiled code. If
box
returned
"OK"
string then test is passed. You can run and debug any of those tests to debug or write you own tests. Just add
myShinyTest.kt
to corresponding testData directory and run
./gradlew generateTests
or
Generate All Tests
run configuration in IDEA, and it will add
testMyShinyTest
to corresponding test runner.
Using FIR compiler for compiling external project Any Kotlin/JVM project may be compiled using FIR (and will be compiled succesfully with high probability). To enable FIR compilation you can just add
-Xuse-fir
to compiler arguments. It's better to use at most fresh compiler as possible, so I recommend you to use compiler which you build by yourself from sources. 1. run
./gradlew publish
. This command will publish all compiler jars to local maven repository at path
kotlinProject/build/repo
2. Add
mavenLocal
repository pointing to this directory to your project 3. Set kotlin version to
1.5.255-SNAPSHOT
4. Add
-Xuse-fir
to
freeCompilerArguments
5. Enjoy You also can attach debugger to process of compiler if you want: - run your gradle task with next flags:
-Dorg.gradle.debug=true -Dkotlin.compiler.execution.strategy="in-process"
. E.g.
./gradlew -Dorg.gradle.debug=true -Dkotlin.compiler.execution.strategy="in-process" build
- create new run configuration in IDEA from template
Remote debug
with port 5005 - run this configuration (in debug mode) after running gradle task Please note that FIR is not completed, so using
-Xuse-fir
is not recommended for production purposes
m

mikehearn

06/02/2021, 12:21 PM
Thankyou Dmitriy, that is very interesting. I have a few questions. 1. Could you elaborate more on the prior architecture and what made it problematic? Specifically, where do the performance improvements come from? All the phases and structure you describe sound pretty conventional. 2. To what extent can things run in parallel with FIR. Is the FIR tree owned by a single thread at once (presumably yes), if so how is parallel compilation managed - is parallelism only possible at the [B]IR level? 3. Although there's not (yet) any public API for writing FIR compiler plugins, how stable are the internal interfaces likely to be? If we want to write changes to the compiler that alter or insert new FIR stages, will there be constant merge conflicts or are the basic internal APIs quite stable now? 4. Why is body resolved twice, once for implicit returns and once for everything else? What's so special about implicit return types?
Background: I'm interested in writing a plugin/altering the compiler to do stuff with string substitutions, but only when the sink of the string substitution is annotated in a certain way. It feels like this is a good fit for the FIR level but it would be a JVM-specific feature, so, maybe it's more appropriate at the back end? I'm not sure if
"""${foo}"""
type strings have been de-sugared by then.
d

dmitriy.novozhilov

06/02/2021, 2:22 PM
1. Main problems of performance in FE 1.0 were related to very bad code locality and memory locality. All descriptors were lazy and because of that compiler always jumps between different parts of code, which kills number of JIT optimization. Also all information about resolution was stored in one big maps of maps (binding context), so CPU cannot cache objects good enough 2. Some of phase I described needs to global resolution (e.g. to resolve supertypes of some class we need to resolve supertypes of those supertypes), and some are very local (to resolve type of value parameter you don't need to access anything else except some provider which can found declaration by it's name). And this local phases may be parallelized (because they don't need have any data contention between threads). Also checkers stage may be parallelized because checkers can not modify the tree, only read data from it 3. FIR internals are not stable at all. Base architecture is mostly stable, but we may introduce some big changes if we decide that it may improve performance 4. Each body is resolved only once. But some bodies are resolved on one phase and other on another.
IMPLICIT_TYPE_BODY_RESOLVE
stage is needed to infer all return types of all functions/properties, which makes
BODY_RESOLVE
stage local (see 2.). On
BODY_RESOLVE
stage when analyzing some function call in some body we don't care is body of this function already analysed or not. We just need types (receiver, parameters, return types). So bodies in this stage can be resolved in parallel (and
BODY_RESOLVE
stage takes ~60-70% of time of all phases)
Background: I'm interested in writing a plugin/altering the compiler to do stuff with string substitutions, but only when the sink of the string substitution is annotated in a certain way. It feels like this is a good fit for the FIR level but it would be a JVM-specific feature, so, maybe it's more appropriate at the back end? I'm not sure if """${foo}""" type strings have been de-sugared by then.
If your changes doesn't affect resolve (and string template will have
String
type after your changes) then it is definitely should be a backend IR plugin. In FIR API we want to give API for providing new declarations (without bodies) and changing existing ones (e.g. change visibility). And filling bodies of generated declarations will be handled on backend side
m

mikehearn

06/02/2021, 3:51 PM
In this case I want to replace String with some other expression that returns a String. Is that still better done in backend?
d

dmitriy.novozhilov

06/02/2021, 7:05 PM
Yes. You can just replace string expression with something yours
m

mikehearn

06/02/2021, 7:07 PM
Backend it is then. Final question - does FIR vs BIR make any difference w.r.t. IDE capabilities or integrations?
Like, if I were to write something for a hypothetical FIR plugin API, would IntelliJ be able to use that for something useful?
d

dmitriy.novozhilov

06/02/2021, 7:37 PM
Yes, we are planning to make such FIR API so compiler plugins will work in IDE out of box
🎉 10
g

galex

10/07/2022, 12:10 PM
Hello @dmitriy.novozhilov , what's the status of FIR? I'm currently struggling to find how to build and test a compiler plugin with FIR to check things and return errors to the IDE (to see red lines under)
d

dmitriy.novozhilov

10/07/2022, 1:01 PM
K2 compiler (FIR) is released in alpha version in Kotlin 1.7.0 K2 Kotlin IDE plugin still is an active development stage and can be build only from sources. For that you need to clone intellij-community repo and run run configuration named IDEA Community (K2 Kotlin)
g

galex

10/07/2022, 1:09 PM
@dmitriy.novozhilov Thanks! Is https://github.com/demiurg906/kotlin-compiler-course-examples/tree/compiler-plugin-project-setup the relevant place to look for an example how to build a K2/FIR Compiler Plugin?
d

dmitriy.novozhilov

10/07/2022, 1:10 PM
It's a good template of K2 project setup (just update kotlin version to smth more fresh, like
1.8.20-dev-649
) For examples of plugin implementation I suggest to check official jetbrains plugins
g

galex

10/07/2022, 7:19 PM
@dmitriy.novozhilov Thanks! A little question about the testing framework and those tests here -> https://github.com/demiurg906/kotlin-compiler-course-examples/tree/compiler-plugin-project-setup/testData/diagnostics Why are there two files,
simple.kt
and
simple.fir.kt
? Is the latter what FIR will generate and that's the instructions for the IDE to show an error?
d

dmitriy.novozhilov

10/07/2022, 7:44 PM
simple.kt contains original code for test, and framework will past warnings and errors directly in it (like highlighting in IDE, yes) simple.fir.txt contains text dump of FIR tree after analysis To add new tests just add new .kt files and run generateTests gradle task
57 Views