Quick question - Is the Kotlin Compiler thread saf...

# compiler

[Jonathan Gerrish]

Quick question - Is the Kotlin Compiler thread safe? It looks like a shared instance of `K2JVMCompiler`is created in KotlinCompileDaemonBase... so I'm assuming so? Background - we are implementing Multiplex worker support for the Bazel Kotlin rules which means a single process will handle multiple concurrent compilation requests so I want to be sure if and how instances should be shared or isolated. Thanks in advance 🙂

[ilya.chernikov]

Yes, the compiler is considered thread-safe, at least if it is used the same way as in the kotlin daemon. But there are many possible caveats there, especially if you want to free resources when there are no active compilation requests. It might be a better idea to reuse kotlin daemon for that purpose.

[Jonathan Gerrish]

Hey @ilya.chernikov - Thanks for the response, we noticed when we switched to handling multiple concurrent requests with the same `K2JVMCompiler`we were seeing sporadic flaky failures in builds which manifested as missing symbols from classpath dependencies. We're switching to a new instance for each request to see if this helps but based on what you said this shouldn't be the problem? I was just wondering if there is any other shared state used by `K2JVMCompiler`that we need to isolate?

[ilya.chernikov]

Sorry, I forgot one important piece: you to set the system property "kotlin.environment.keepalive" before calling compiler.
The K2JVMCompiler class contains little state itself. All possible sharing is happening behind the scenes. So probably single instance will work with the property I mentioned above, But it is safer to create a new instance for each compilation. At least this is the scenario that we're using and therefore testing against.

[Jonathan Gerrish]

OK, thanks that's good to know, we're running a separate builder to monitor for these flakes... if switching to a single instance solves the issue then we can switch over.

[Jonathan Gerrish]

...and a random question @ilya.chernikov - we're using a compiler plugin for dependency tracking in a module, recording declaration and call references but its a bit tricky to get right, I just noticed these compiler services, such as LookupTracker is this a better way of recording what dependencies are loaded by the compiler for a particular invocation?

[ilya.chernikov]

It depends very much on your usage scenario. We are using lookup tracker to detect dependencies for incremental compilation, but it is only a part of the whole IC machinery, e.g. there is a separate entity to track Java classes. So if this sounds like your scenario, you may try to use LookupTracker. Just be careful if you decide to modify it or its use sites - these are quite hot places and doing so it is quite easy to degrade the performance.
And, I hope you understand this, but these are internal compiler APIs so we're spending no efforts to maintain their stability and so on, so it might be better for the stability of your software. to avoid using it, if there are alternatives.

[Jonathan Gerrish]

Hey @ilya.chernikov - sorry one last thing, can the system property `kotlin.environment.keepalive`be considered as a stable part of the API... that is, if we use it in the Bazel rules (and stick with a singleton compiler) is it likely to change? The safest thing as you mentioned is going to compiler per request. Thanks in advance!

[ilya.chernikov]

Hi Jonathan.
Just to avoid misunderstanding, you need this property for the compiler per request too. And you probably will not save anything performance-wise with a singleton compiler.
As of the stability - we have plans to rewrite this part along with introducing the new compiler frontend, but it will take a year or more until the new implementation will become default.
But the current one is internal anyway, so we're not checking external dependencies when doing changes in these parts, so we could break it accidentally nevertheless. I don't know whether it makes sense to do anything to prevent any breaking changes there, I personally think that the probability is quite low. But maybe it makes sense to include some Basel-based project to our test set.
And I hope we'll have enough time to provide more "official" way to use the compiler as a library, along with the new internal API and the new frontend.

[Jonathan Gerrish]

Thanks @ilya.chernikov - here is the PR to make this change for Bazel's Kotlin worker https://github.com/bazelbuild/rules_kotlin/pull/515 - if you'd like to include a Bazel based project in your test set I think a good one would be to just use the Kotlin Rules themselves, that would be really helpful and we'd definitely be responsive to anything that comes up for you! 🙂