Hey compiler plugin developers. I am curious to ga...
# compiler
t
Hey compiler plugin developers. I am curious to gauge the following idea with the following rough outline - We are thinking about introducing compiler plugin devkit in a form of separate Gradle plugin. The core idea is that it will create a set of the following source sets: • API - common api for the compiler plugin (for example compiler plugin options) • implementations for concrete Kotlin compiler version. The supported versions are defined in the DSL and will be provided as
compileOnly
dependencies by devkit plugin • optional bootstrap for Gradle and Maven plugins The project will publish a set of publications for API + implementation for every supported Kotlin compiler version with additional (Gradle or some way) attribute. Based on this attribute Kotlin build plugins could select proper publication for used in the project Kotlin compiler version. What do you think about it?
🔝 3
o
Currently, TestBalloon publishes a series of artifacts per its own version combined with all supported Kotlin compiler versions. In most cases, the compiler plugin is the only part that's different. If I understand correctly, that would allow me to distribute one plugin artifact under one version, which would automatically work with all compiler versions. That sounds very useful indeed!
y
Would this mean that it compiles against all the Kotlin versions? I don't see how error-highlighting in IDE would work, for instance. Also, I don't like being limited to a specific set of compiler versions because EAP versions can be useful to try out sometimes, or even
dev
versions
t
supported compiler versions developers will specify in devkit plugin DSL. You could also specify non final release versions
c
I hate to be a newbie who acts like they know everything, but I do genuinely have some experience in this specific case. As someone who's worked in a similar field - the modding framework for Minecraft* - binding the modding framework to Gradle always seems like a good idea until you realize how much it restricts on the user end. Gradle's task system makes the backend of any plugin's "expected build chain" a black-box, not allowing users to inject in the middle of two interlinked tasks, or even inspect the full task graph in most cases. If the user needs to apply their own preprocessing to the output jar or has a non-standard (but still reasonable) module system, it can be impossible to insert their own necessary tasks through the tightly-bound task flow. Binding things to specific source-sets can also easily cause issues with any kind of "off the beaten path" module setup the user might genuinely need for their specific use-case. If you do decide to follow through with this, you also need to take care that the dependencies used aren't tightly-bound to Gradle itself, thus making any plugins built with it incompatible with non-Gradle build systems. Overall, even though it's "easier" to bind the process to Gradle, it's far better to keep things open to the user in the form of convention plugins in a shared build template. That way the user can actually tweak things if needed, instead of being walled off from the backend with a proper plugin. [*Don't knock it, it's a larger task than most corporate projects I've worked on. We have to deobfuscate the original binary; remap both the original files and each third-party dependency through two different intermediate mappings so it's easy to migrate mappings between game versions; decompile the deobf jar with custom formatting, mappings, and documentation so the user can read it; reconcile both unmapped and fully-obfuscated dependencies; then fully transform the output jar to allow it to run with the obfuscated Minecraft jar; ALL without actually caching anything so we don't violate copyright.]
o
Isn't that attribute matching something which KMP already does?
➕ 1
h
Sounds very good to me, if I can develop a compiler plugin using multiple versions at the same time, like KGP itself supports multiple Gradle versions by using multiple sourceSets/java features. So consuming it via 1 version should be possible
t
KMP does resolution based on Gradle attributes, but we don't have an attribute for Kotlin compiler version
So consuming it via 1 version should be possible
Yes, this is the main idea here - to have one compiler plugin publication supporting a range of Kotlin compiler versions
➕ 2
o
That's what I intended to say: @Caleb B seemed to assume that the technique was novel to the Kotlin compiler landscape, but actually it is not. So it's just the established pattern applied in a different way.
h
Nice, but then also support this with IntelliJ for FIR plugins, by overriding the attribute value with the builtin Kotlin version during sync.
➕ 1
t
yes, we could have a variant for Kotlin compiler version bundled into IDEA and resolve proper one on import
c
@Oliver.O It's not novel to the landscape, but it is shortsighted. I can't count the number of issues I've had with trying to make KMP work for non-standard use-cases. It's better to let the user tinker with the internals if need-be instead of making the backend immutable. Tightly binding it to Gradle is what makes KMP completely unusable for multi-JVM, embedded, and other non-standard -target projects.
t
@Caleb B we acknowledge that KMP currently is tightly coupled with Gradle and we are working on improving situation here
though this topic is not about KMP per-se
c
I'm glad to hear it! I'm just saying that as someone who's implemented an extremely similar system for a different project, it's basically impossible to make a plugin that isn't tightly-coupled - both to Gradle and to the "single expected use-case" (however wide that may be) - just because of how Gradle's plugin system and task graph works. Once something is set by a plugin, there's no way to override or veto a choice on the user end; you can only either interface with it as intended or get rid of it altogether. If we were to implement our system again, even though it's insanely complex, we'd still put the logic in the buildSrc of the SDK instead of distributing it through Maven, so people are able to tweak it to their needs if they need to venture off the beaten path. Anyway, I'll stop clogging up the channel. I just wanted to give a warning from someone who made the same mistake.
I do love the shared set of attributes though. That's really useful, and that does belong in a published plugin instead of the SDK.
j
Love the idea ❤️
m
> The project will publish a set of publications for API + implementation for every supported Kotlin compiler version with additional (Gradle or some way) attribute. I think I'd like it to be a single .pom publication without Gradle attributes, same as uklibs. I'm a bit wary of the proliferation of Gradle attributes and publications. A single file and
kotlinc
would know how to unpackage it. A bit similar to a multi-release jar file.
➕ 1
z
Love the idea in principle. One concern I'd have would be how testing would work for such a scenario (especially with the compiler test infra)
➕ 3
r
I like the idea. Would the publishing stage (and/or attributes) help with needing to only publish shaded versions so that the native compiler picks them up? Also, including the compiler test setup logic could be a big help, too.
t
regarding the testing support - it is a good point and we will consider it
Created an issue for this idea: KT-82617 DevKit for Kotlin compiler plugins
kodee happy 1
y
A lot has been said about the potential for this dev kit being too tied to gradle and not configurable enough and such. I want to argue a bit on the other side of that. Right now, we're writing plugins with no such devkit support. If the devkit helps even 50% of compiler plugin cases, then it's a win in my eyes. As long as the team continues to support the "old" way of defining compiler plugins, i.e. without the devkit and doing things manually, then this'd be a net win
➕ 2
The example of Minecraft modding brought up is an interesting case study and should definitely be taken into account when designing this; however, there's one crucial difference here: the same team making the language is the one designing the modding API and tooling support. Thus, a lot less tinkering and magic needs to happen to make things work. The Kotlin compiler is open source, Minecraft isn't AFAIK. A more apt comparison is the IntelliJ Platform DevKit, which is incredible. It does multiple version testing and all of that for you. Yet it's still configurable enough.
m
dev kit being too tied to gradle
FWIW, I don't mind a Gradle devkit. People use Gradle and it's great to support them with a Gradle plugin that makes it easy to support multiple
kotlinc
versions. What I'm not a fan of is using Gradle metadata as the interchange format when it's not clear if it brings any value while making it harder to interop with other build tools.
z
I’m somewhat ok with the gradle metadata given the build-system-agnostic kotlin build tools initiative that would presumably cover this for other build systems
👀 2
y
@Zac Sweers any links for info on that initiative? I wasn't aware of it, but it sounds really interesting
z
I’m on mobile but it was announced in the kotlinconf keynote
thank you color 1
t
Build Tools API?
z
Yeah
t
m
Even if it becomes 100% an implementation detail, the choice of Gradle metadata as the file format feels a bit weird. I don't expect the Build Tools API to replicate the Gradle resolution algorithm for an example.
t
No, BTA will not replicate it.
👍 1
z
oh in that case, I understand martin's concern
t
Build Tools API should not do any resolution or provisioning of things
z
I had assumed it was going to offer something in terms of resolution for KMP already
t
consider it as an API for Kotlin compiler ecosystem
👍 1
we have a separate discussion track regarding KMP, Gradle metadata and UKlibs, but it is a hard topic and there are many variables
m
The main (only?) advantage of Gradle metadata is saving bandwitdth for Gradle users because you can have Gradle parse the file format at download time.
But that comes with a very high cost in terms of complexity
c
But yeah sorry for the edit spam y'all, I didn't realize
o
Just edit outside next time, then copy paste. 🙂
👍 1