Hey folks! :wave: My name is Eugene, and I'm a pro...

# build-tools

[Eugene]

Hey folks! 👋
My name is Eugene, and I'm a product manager at JetBrains.

In build-tools, 🛠️ we are seriously thinking about the roadmap and wondering what things might be considered as critical from your perspective. So, I have an ask for each of you.

📢 Please, let me know in the thread/privately what is important for you❗*.*

It might be issues, pull-requests, feature requests, or even fundamental problems. This information will help me to focus my research. And hopefully, your issues become a part of the backlog.

Thanks for your attention and happy coding)

[mbonnin]

Not sure if it falls into #build-tools but my biggest pain is with IDE support for MPP projects at the moment. The developer experience is far from optimal there. This bug in particular makes developing for Android + MPP way harder than it should be: https://issuetracker.google.com/issues/165921901

[mbonnin]

Beside that, faster builds are always nice!

[Joe]

1. jacoco coverage for inline functions: https://github.com/jacoco/jacoco/issues/654
2. don't abandon maven in favor of gradle (understanding that gradle may be preferred)

[Marc Knaup]

Hey @Eugene 👋

I’d add a more reliable Gradle integration as well as Kotlin Gradle Plugin improvements.

For example a Gradle sync is mysteriously successful even if a dependency is missing.

And many kinds of exceptions are properly presented in CLI but not in the Gradle sync window. One has to guess what’s happening if a sync fails. Sometimes there’s no output. Sometimes it’s reported as successful but it actually wasn’t. Sometimes it states there is an exception but refuses to give me details.

And then there are multimodule multiplatform projects with composite builds. There’s basically something different wrong with almost every Kotlin release 😅 I consider composite builds vital because libraries are often developed along software that build on them. I need to have a rapid development<->test loop which is not possible if I have to republish every single change in the library to Maven Local.

Stopping a running Gradle build from IDE also doesn’t always work. Especially when the build is part of a run action. It will run no matter how often I cancel.

Running tests for entire folders is broken because the IDE assumes the folder name is actually the package name. But that’s not necessarily the case in Kotlin.

When running tests from IDE and there are build failures in the test files, I’ll get an error that “no test events have been received”. That’s confusing. If I remember to scroll down the entire output window I may notice that the build failures are hidden there. Why isn’t that clearly visible in the Build tab?

Oh and configuring Kotlin projects, esp. when there are multiple modules and/or platform, is ridiculously complicated.
I went so far to standardize all my Kotlin libraries with my own Gradle plugin (https://github.com/fluidsonic/fluid-gradle).
For non-library projects I have yet to add something like that. I keep learning new subtleties about Gradle configuration or things to make things work more reliable or to write something more concise. Even after years. And I keep piling up workarounds for issues. Like publishing…

Publishing Kotlin projects, esp. multiplatform, to Maven is really annoying. Maven publishing itself ain’t easy, but with multiplatform there are so many modules and files to be uploaded that configuration and reliability become an issue. Ktor itself for example has a script that retries uploading every single file to Bintray 100 times 😮 There should be a better and more efficient way to publish Kotlin libraries.

Setting up Gradle projects is basically a huge amount of copy & paste for me. And more copy & paste whenever something in Gradle or Kotlin’s plugin changes.

A quick & easy way to debug Gradle syncs including buildSrc would also be great. Problems anywhere in the Gradle sync inevitably happen. Often.

Another example from yesterday:
I wanted to enable explicit API. I was testing various approaches, googled and looked at Kotlin Gradle Plugin sources to figure out where in the way too large build.gradle.kts I have to call this. Turns it it’s in the kotlin block and not in kotlinOptions or languageSettings where I’ve expected it.

And there’s also the mix between Groovy Gradle files and Kotlin Gradle files. KTS is quite advanced by now yet many Kotlin projects and samples still use the Groovy syntax. Instead of that, JetBrains projects should be dogfooding Gradle KTS. That also helps people learn. Due to lack of good documentation and samples it’s difficult to understand how to do a wide variety of things in KTS.

[mbonnin]

++ for composite and multiplatform.
Publishing errors are mostly from Bintray. Although, I would love Jetbrains to offer a good/reliable package manager with a good UI 🙂

[Eugene]

@Marc Knaup fantastic, thanks for these details! There really is something to think about. Maybe a bit later I'll drop a note to you to discuss some of the items above.

[Marc Knaup]

👍

Do you consider debugging in IDEA as part of “build tools”? Or even the entire IDE? I’m not sure where you draw the line 🙂

If it’s part then please add debugging issues to the list.

I’ve spent hours yesterday debugging with println because the debugger was not helpful. If there are many coroutines then the Coroutines view basically loads forever and is unusable. In some cases stepping through code takes minutes, as does loading variable values. And very often breakpoints are simply ignored, esp. when coroutines are involved. Stack traces with coroutines are not helpful either.

[Eugene]

Marc, are you talking about AS or IDEA?
BTW: debugging is not a part of build tools

[Marc Knaup]

IDEA Ultimate

[Eugene]

Could you share YT issue? I'll clarify what's state of this problem,

[Marc Knaup]

For the coroutines view I did this night: https://youtrack.jetbrains.com/issue/KT-42594

For the other ones I will as soon as I encounter them again.

[Eugene]

No rush! And again, thanks for your feedback

[Marc Knaup]

Here’s a crazy idea 😄
Open a Slack channel like #venting where developers can easily vent about any frustration they have right in this moment. It’s way faster and easier to give and get feedback than having to create a YouTrack issue or sending an email.
It also gives an idea about how frequent an issue occurs.

[mbonnin]

We have this in the francophone Android slack because we like to vent a lot I guess 😅

[CLOVIS]

Not sure if JetBrains can do anything about the publishing part, but I never understood how to publish a Kotlin Multiplatform artifact to MavenCentral, or even GitLab (through Gradle).

I must say though, publishing a normal JAR on Space has been a breeze!

[Zac Sweers]

• IDE indexing time remains a historically painful issue, early testing of 1.4.20-M1 are promising.
• Build times (particularly FIR) will be big. 4-5x improvements sound almost too good to be true, will be very curious to see if that holds up
• Retire kapt and consider fully endorsing KSP. Kapt appears largely unmaintained and it kind of hurts trust with both it and the rest of the build tools as it often acts as a major bottleneck
• Figure out incremental compilation's relationship with compiler plugins. Anvil, KSP, and likely others all require disabling incremental compilation right now to use and don't have much of an obvious recourse
• Stop using absolute paths in gradle tasks, as they break build caching. This has been marked as "major" priority but has seen no movement in a year, and results in teams having to disable kotlin build caching entirely.
• Support source-based ABI jar generation. The current ABI jar generation effectively just runs the compiler twice, making it unusable and sort of negating its purpose of being a build performance optimization tool
• Better support bazel
• Discontinue support for Maven and move any resources for it into gradle or bazel. The maven plugin appears to be barely maintained as-is, seems like it would be best to wind it down
• Don't publish artifacts solely to jcenter. Only Dokka seems to really have this issue, but it's a security risk for developers.

[Zac Sweers]

Can't edit the above but I'd add that kapt contributions appear to rarely get reviewed, adding to frustration

• https://github.com/JetBrains/kotlin/pull/3752
• https://github.com/JetBrains/kotlin/pull/3610

[basher]

I’ve been following up on this thread and related tickets, and it seems like friend modules are maybe supposedly implemented for MPP in Kotlin 1.4.0? Is that accurate? In the gradle plugin code, I still see that associateWith restricts to a single target. Is there another DSL we should be using to setup friend modules? Would love to see that work for MPP. We’re getting started modularization for our main library and have realized how impactful this would be

[h0tk3y]

Hi @basher,
Associated compilations are indeed a close equivalent of friend modules, but yes, they can only work inside a single target (therefore, only inside one module).

Does your use case assume setting up friend/internal visibility between different projects in a multi-project build? I'd really like to learn more about your use case or take a look at the project if it's open source.

Basically, supporting friend modules across projects might make builds more fragile, because references to internal members might break on changes in the referenced modules that are not even changes in the public API (and thus normally retain binary compatibility), so it might not be obvious that the depending module has to recompile after those changes. We will have to carefully consider the risks and potential pitfalls. However, as most builds assemble and publish all of the modules at once, it doesn't seem to be a critical issue.

One more possible solution that I can think, which however requires a different architecture, is this: you can put the internal APIs in a separate project that you add to the depending modules as an implementation dependency. This won't bring the module with internal APIs to the consumer's compilation classpath and will only bring it to the runtime classpath. It's more or less equivalent to introducing friend dependencies, because in both cases you shouldn't use the internal declarations in your public API.

However, I can imagine how having friend modules could be more convenient than moving the internal APIs to a separate module that is added as an implementation dependency, and we should definitely consider this based on the use cases.

[Marc Knaup]

I had the same issue and worked around it using @RequiresOptIn to make internal API unusable by default. It can also break if multiple modules that rely on it have different versions but I kinda expect that. Friend modules would just be a way to formalize such an approach, wouldn’t it?

https://github.com/fluidsonic/fluid-graphql/blob/master/modules/language/sources/common/utility/InternalGraphqlApi.kt

[basher]

Does your use case assume setting up friend/internal visibility between different projects in a multi-project build

I think the answer to this is yes, though I’m new to composite builds and the associated terminology.

We have a project that has become quite large, which we compile into a single library and is consumed by some downstream projects (in other repositories) internally at Autodesk. We’ve been careful to use explicit API mode, so visibility is well-defined for all of our APIs. Most of our APIs are marked as internal whenever possible, so that we can reap those benefits of avoiding breaking downstream clients.

Now that the project is large, we’d like to breakup the project into sub-projects/modules to make better use of gradle/compiler caching and parallel compilation. Right now, because the library is all one big module, all of the library classes use other library classes internal API, and that is okay and is something we’d like to continue to preserve. If we split apart the library into modules, we’ll have to make more classes and methods public, so that other modules within the library will be continue to be able to talk to each other. The main issue with this is that because we’ll have to now publish all of these modules in concert to continue to allow JVM clients to consume this library, there’s nothing stopping JVM clients from using internal APIs that we had to make public to make modularization possible within the library.

@RequiresOptIn is one avenue, but KMP makes this more complicated. It’s unclear (or at least not well-documented) whether those annotated APIs would be exposed to K/N clients, if we needed to export one of those modules to make some public APIs from those modules available in an iOS framework, for example. All of that forces us into using naming conventions like util and util-internal where the util module is okay to export or use as an api dependency, where util-internal should never be exposed publicly.

I hope this clarifies our situation. Thanks again!

[h0tk3y]

Thanks for the input! It's a curious point about declarations marked with @RequiresOptIn being eligible for export in iOS frameworks, I think I'll pass this to our Kotlin/Native team.

Supporting friend dependencies is definitely a thing for us to consider, although it requires thorough design, which may take time and uncover other problems.

One more potential solution that I see is that it should be possible to customize the build so that it post-processes the output artifacts that are published for the internal API modules (or even more granular, just the internal packages) in a way that makes them inaccessible for other Kotlin consumers outside of your build, while your build could still use the original artifacts. For example, it should be possible to add @Deprecated with level HIDDEN to the declarations. This doesn't seem to be anything simple, though, given the multiple artifact formats that you might want to post-process – JVM bytecode, *.js files, and `*.klib`s. Maybe writing a Gradle plugin that does this is a good idea.

[basher]

It’s a curious point about declarations marked with @RequiresOptIn being eligible for export in iOS frameworks, I think I’ll pass this to our Kotlin/Native team.

Maybe it’s already handled? I’m not sure, but I would expect that it’s not. But sure, I look forward to hearing what K/N folks have to say 🙂

In any case, a solution that requires an annotation to hide APIs from the public brings with it a lot of overhead:
1. Actual overhead of properly annotating all of your APIs
2. Mental overhead of “is this public? oh no, it’s annotated with @X
At that point, how far are you really from adding package private to Kotlin? I’m not suggesting that package private is the solution, but the level of condoned hackery begs the question I think 🙂

[basher]

In any case, if @RequiresOptIn were honored by the K/N backend when exporting a module, that might be sufficient.

[h0tk3y]

Just got the answer from the Kotlin/Native team that @RequiresOptIn APIs are exported in frameworks as normal public API. They haven't considered limiting this as of yet.

[basher]

Is there a ticket yet to support it?

[h0tk3y]

Sorry for the late reply, but no, there's no ticket to support that yet, feel free to submit one.

[basher]

https://youtrack.jetbrains.com/issue/KT-43727

[h0tk3y]

Thanks!