Hi, the system I'm currently working on generates a set of Kotlin source codes on the fly and compil...

# compiler

[dhkim]

Hi, the system I'm currently working on generates a set of Kotlin source codes on the fly and compiles them using K2 compiler. It's done at runtime and it's important to keep the whole time required from the source code generation to compilation minimal. In the light of minimizing the time, I'm about to explore uses of FIR plugins. If I generate FIR in my plugin instead of the comparable source code, how much time can be slashed? Like what percent of ANALYZE metric obtained from compiler performance report. Just a ball park figure would be appreciated.

[dmitriy.novozhilov]

Generally speaking generating fully analyzed code is faster than passing raw sources, because the compiler won't need to analyze the code by itself, but believe me you don't want to generate fully analyzed code manually.
• It would be extremely complex task, as there are a lot of information you need to provide to FIR/IR nodes which is quite untrivial to calculate, especially if you don't know all the details
• You'll face the need to update/rewrite your code with every compiler update, as there are no compatibility guarantees for compiler internals

[dmitriy.novozhilov]

Do you just invoke the compiler each time (and how do you do that)?
If yes, there is a room to optimize the whole process in much simpler means

[dhkim]

Thanks for the information. I use Kotlin-compiler-embeddable and keep the server running at the time. I think it should be similar to Kotlin playground.

[dhkim]

Even with the above information, I'm determined to take another route. 🙂

[dhkim]

Could you bother to tell me the simpler means you mentioned above?

[dmitriy.novozhilov]

keep the server running at the time

That was my idea, actually

[dhkim]

Ok. Besides, I'm trying to generate source codes in a way each file can be compiled separately. Thus I can implement simple incremental compilation without hacking the Kotlin gradle plugin and K2 compiler.

[dmitriy.novozhilov]

First run of the compiler is always slow, as
• there are some static state which is need to be initialized
• the code of the compiler is not optimized by JiT
So keeping the compiler alive between invocations helps a lot. But I already do it

[dmitriy.novozhilov]

without hacking the Kotlin gradle plugin and K2 compiler

What hacking do you mean?

[dhkim]

Thanks a lot. Your explanation saved a lot of my time.

[dhkim]

Hacking? I still don't know much about how incremental compile works. I need to look into the Gradle plugin and K2 documents to understand and leverage it on my project.

[dhkim]

(Maybe hacking isn't a good word for describing what I have in mind. Sorry)

[dmitriy.novozhilov]

Also compiling two files together is faster than compiling them one by one.
Compiler doesn't preserve loaded dependencies from compilation classpath between compilations, so if, for example, both of your files refer to some class Foo from dependencies, it would be deserialized only once if files compiled together and twice if compiled separately

[dhkim]

Ok, I got it.

[dmitriy.novozhilov]

Why do you care about incremental compilation at all if you generate completely new files each time?
IC is needed mostly for cases when you have one big module and few files changed. So with IC you will compile not the whole module, but only changed files and ones which resolution could change because of these changes

[dhkim]

Though I generate the whole set of source files every time, some of them will be unchanged between generations since generation is driven by end user.

[dhkim]

One tick in the UI may change a small part in the generated code.

[dmitriy.novozhilov]

Ah, in this case it's important

[dmitriy.novozhilov]

Did you try a more simple setup without need to call any internals?
You can setup a gradle project with empty src folder, generate new sources inside its src and then just call the gradle to build it (starting the ./gradlew process).
In this case Gradle and KGP will work as for regular project

[dmitriy.novozhilov]

One tick in the UI may change a small part in the generated code.

Oh, it's about UI. In this case everything should be really fast

[dmitriy.novozhilov]

@Sebastian Sellmair [JB] could the Hot reload somehow help here?

[dhkim]

Hmm. I've ever heard about it, but never thought it might be helpful in my setup. Should I start at https://github.com/JetBrains/compose-hot-reload ?

[dmitriy.novozhilov]

I'm not sure if it will be applicable, that's why I called Seb.
But it anyway makes sense to check this repo, maybe you'll get some idea

[dhkim]

Thank you for your valuable time. Every piece of information you gave above is helpful.

[Sebastian Sellmair [JB]]

Hey 👋
Yes, using Compose Hot Reload should be a nice way to actually prototype:
You could generate new sources and then let the build system compile incrementally, updating your code live.

However, if you would like to generate source code within a process and load that, then maybe you might want to take a look into some test here:
https://github.com/JetBrains/compose-hot-reload/blob/24f4f3b735b15d59e8dd8fd38afcc4f390b4792f/tests/src/reloadUnitTest/kotlin/tests/enums.kt#L15

There is a 'compileAndReload' function that might be nice to look into:
https://github.com/JetBrains/compose-hot-reload/blob/master/hot-reload-test/src/main/kotlin/org/jetbrains/compose/reload/test/compileAndReload.kt

Without tests & hot reload in the picture you could actually just get the bytecode from the compilation and call
instrumentation.redefineClasses (see example: https://github.com/JetBrains/compose-hot-reload/blob/24f4f3b735b15d59e8dd8fd38afcc[…]nt/src/main/kotlin/org/jetbrains/compose/reload/agent/reload.kt)

[Sebastian Sellmair [JB]]

Also, if you just always want to compile new classes, then just using MethodHandles to define a class might work for you!

[Sebastian Sellmair [JB]]

Generally, I would expect your approach not to benefit a lot from IC, but this depends a lot on "what you actually generate" at runtime

[dhkim]

Thank you. I'll take a look at what you listed above.

For the IC part, I've run a rough test and seen some meaningful reduction in the total generate/compile cycle time. But I think I'm not taking the full IC route. Instead we will pick files to recompile based on far simpler criteria of just picking changed ones (we need to keep the generate files in a way that such an assumption is valid).

Similarly we originally employed KSP2 to generate codes required for our internal framework. But due to the base overhead incurred by KSP itself, we turn to "read YAMLs and generate codes" approach and slashed a good amount of time from the whole generate and compile cycle.