https://kotlinlang.org logo
#gradle
Title
# gradle
a

Andrew Louis

02/22/2023, 2:28 PM
👋 Hey folks, I'm trying to understand how I can troubleshoot v slow
compileKotlin
task times -- it takes up to a minimum of 3 minutes each time a Kotlin file is changed in the main sourceset 😢 • Build caching is enabled -- with a Gradle build cache node in use (cache is fresh and definitely getting hits, used the cache debug flag to verify) • Heap alloc is pretty high ◦
org.gradle.jvmargs=-Xmx8g -Dkotlin.daemon.jvm.options=-Xmx6g
• It's a single module project @ around 80k LOC Please let me know what some options are to try and troubleshoot the task closer -- I tried using the profile & scan flags, but it didn't really break down what the task was spending its time doing
j

Javier

02/22/2023, 2:35 PM
Have you tried to run check a build scan?
a

Andrew Louis

02/22/2023, 2:35 PM
Yessir
Didn't really reveal why the compileKotlin task itself was taking time; it was useful to identify that
compileKotlin
is the longest running task
v

Vampire

02/22/2023, 2:49 PM
You could have a deeper look using https://github.com/gradle/gradle-profiler probably
But maybe the Kotlin compilation simply needs that long
e

ephemient

02/22/2023, 2:51 PM
if it's a single module, it should be doable to build up a argument list to run the command line
kotlinc
for comparison
but that sounds incredibly long for such a small project, are you doing anything like https://youtrack.jetbrains.com/issue/KT-41965 that could be causing non-linear behavior in the compiler?
a

Andrew Louis

02/22/2023, 3:11 PM
Bjorn -- not sure if that profiler gives more insight into the compilation step, but will be useful to make changes and profile to see if it improves the situation Ephemient -- The
kotlinc
idea is pretty interesting, didn't consider it -- can definitely give it a stab for reference. That was an interesting thread to read; I have no idea how that author narrowed it down to that method -- he doesn't seem to discuss it there I just checked, and we don't have any implementations of Comparable that we do ourselves
a

Adam S

02/22/2023, 3:49 PM
what happens if you comment out the ktlint build config? So there's no ktlint interfering at all And how much do you use Lombok?
t

tapchicoma

02/22/2023, 4:45 PM
if it's a single module, it should be doable to build up a argument list to run the command line kotlinc for comparison
It is possible to get them from debug logs
generally if you could provide a repro - I am interested to look into it
8Gb for Gradle and 6Gb for Kotlin is too high for 80 LOC project 🤔
From build scan I see Gradle daemon consumes ~300Mb. Kotlin daemon memory usage should be printed in Kotlin daemon logs. Soon such metric should be available in build metrics produced by KGP
btw @Andrew Louis could you enable build reports and share them?
a

Andrew Louis

02/22/2023, 4:52 PM
It is possible to get them from debug logs
Oh nice, will try it and report back later today Re - build reports - yes, will do! Let me enable it and share shortly Regarding a repro, not sure if this is possible at the moment, but will loop back. Can do an attempt at generate something minimal, but I'm not confident I'll be able to do an open repro (company codebase) 🤔 Will have to look at some options here and get back to you
t

tapchicoma

02/22/2023, 4:53 PM
please share at least a build report - it may lead investigation into proper direction
a

Andrew Louis

02/22/2023, 5:29 PM
@tapchicoma built with the
build_scan
report output: A couple of weird observations: • There's something pretty funny there --
Connect to Kotlin daemon: 47365ms
?? • Also, it's flagged as
NON_INCREMENTAL
blob thinking upside down
t

tapchicoma

02/22/2023, 5:31 PM
NON_INCREMENTAL
could be because you run
clean
before or changed build classpath
a

Andrew Louis

02/22/2023, 5:31 PM
Gotcha, let me try again and see if that goes away
t

tapchicoma

02/22/2023, 5:38 PM
Could you try to run the build on non-GraalVM JDK?
a

Andrew Louis

02/22/2023, 5:38 PM
👍 will do
Ah the last one was non Graal (see link) ; I believe I was trying out Graal temporarily to see if it improved the situation
t

tapchicoma

02/22/2023, 5:39 PM
yeah, accidentally looked into previous build scan
just in case - could you also try to run the build with JDK 17.0.6?
and are you using native Mac/ARM JDK?
a

Andrew Louis

02/22/2023, 5:50 PM
Copy code
openjdk version "16.0.2" 2021-07-20
OpenJDK Runtime Environment Zulu16.32+15-CA (build 16.0.2+7)
OpenJDK 64-Bit Server VM Zulu16.32+15-CA (build 16.0.2+7, mixed mode)
I believe so
And yeah, let me just generate one last set of build scans, and then will try 17.0.6 👍
a

Alexander.Likhachev

02/22/2023, 6:01 PM
If you invoke
java -XshowSettings:properties -version 2> >(grep "os.arch")
in terminal, will it print
os.arch = x86_64
or
os.arch = aarch64
?
a

Andrew Louis

02/22/2023, 6:01 PM
aarch64 -- makes sense for an M1
a

Alexander.Likhachev

02/22/2023, 6:02 PM
I’m asking because you could use both of them on M1 Mac, but the one built for
x86_64
is expected to be slower
a

Andrew Louis

02/22/2023, 6:02 PM
Ah makes sense -- seems like I'm using the arm one
v

Vampire

02/22/2023, 6:04 PM
Side question, what advantage do you see with using
>(grep ...)
instead of
| grep ...
?
a

Alexander.Likhachev

02/22/2023, 6:05 PM
Java outputs the properties into stderr
e

ephemient

02/22/2023, 6:06 PM
2>&1 | grep
then
v

Vampire

02/22/2023, 6:06 PM
Oh, that
2> >(grep ...)
feeds fd 2 to the subshell? Interesting. I would have done
2>&1 | grep ...
. TIL, thanks
e

ephemient

02/22/2023, 6:08 PM
it works but the shell job doesn't include the
>()
process substitution so sometimes its output will be delayed until after the next terminal prompt has already come up :-/
v

Vampire

02/22/2023, 6:09 PM
ah, good to know, thx
a

Andrew Louis

02/22/2023, 6:16 PM
@Yahor - with JDK 17, seems like a big boost - 1 minute 30 on the first run of
compileKotlin
https://scans.gradle.com/s/uajg3nzz2paug
Also I missed a dangerous typo up above; I meant
80k LOC
not
80 LOC
😂 in case that threw anyone
v

Vampire

02/22/2023, 6:27 PM
It for sure did
e

ephemient

02/22/2023, 6:27 PM
ohhhh that makes a big difference :D
v

Vampire

02/22/2023, 6:27 PM
Not me, as I didn't even read it, but yeah, others referred to 80 LOC 😄
a

Andrew Louis

02/22/2023, 6:30 PM
Hahah 💀 apologies for the confusion folks, updated counts after an espresso: main source set: 47k LOC codegen generated kotlin files: 90k LOC
Is the 2-3 minutes within reason in that light?
t

tapchicoma

02/22/2023, 7:14 PM
build with JDK 17 looks much better.
Is the 2-3 minutes within reason in that light?
For clean build I would say yes
r

rrva

02/22/2023, 8:10 PM
Tried these? • Use the k2 compiler We have been running k2 for a kotlin/jvm project for a long time locally and then we disable it in CI. A nice tradeoff if you do not want to use it for release builds
Copy code
val isCiServer = System.getenv().containsKey("CI")
Copy code
tasks.withType<org.jetbrains.kotlin.gradle.tasks.KotlinCompile>().configureEach {
    kotlinOptions.useK2 = !isCiServer
}
or in 1.8.20+
Copy code
tasks.withType<org.jetbrains.kotlin.gradle.tasks.KotlinCompile>().configureEach {
    kotlinOptions {
       if(!isCiServer) {
           languageVersion = "2.0"
       }
    }
}
• Use the incremental compilation support, it will be default in 1.8.20 https://blog.jetbrains.com/kotlin/2022/07/a-new-approach-to-incremental-compilation-in-kotlin/ • Use the multithreaded compilation mode -Xbackend-threads=0
s

semoro

02/22/2023, 10:00 PM
Andrew Louis [8:16 PM]
@Yahor - with JDK 17, seems like a big boost - 1 minute 30 on the first run of
compileKotlin
Please use JDK built for aarch64, not x86 You are running it through Rosetta, which affects performance noticeably
a

Andrew Louis

02/22/2023, 10:02 PM
Hmm, how could you tell I was running it through Rosetta?
s

semoro

02/22/2023, 10:06 PM
Probably I'm wrong, JDK version in the build scan doesn't report JDK arch, but rather 64-Bit Server regardless of the architecture 😞
t

tapchicoma

02/22/2023, 10:09 PM
@Andrew Louis if possible could you also produce build scan with build report while using Kotlin 1.8.10 in the repo?
a

Andrew Louis

02/22/2023, 10:09 PM
Confirmed that I'm using the aarch64 JVM, the
Kind
column in Activity Monitor shows Apple for my java processes
Yessir will do Yahor
Hmm not much change on the initial/clean build: https://scans.gradle.com/s/36gghtzscpk52/custom-values On the incremental, seems much snappier than the 1.7.x https://scans.gradle.com/s/36gghtzscpk52/custom-values
t

tapchicoma

02/23/2023, 9:32 AM
@Andrew Louis yeah, clean build looks slightly worth. Could you also try to run clean build with Kotlin 1.8.20-Beta and then additionally enable K2:
Copy code
tasks.withType<org.jetbrains.kotlin.gradle.tasks.KotlinCompilationTask<*>>().configureEach {
    compilerOptions.languageVersion.set(org.jetbrains.kotlin.gradle.dsl.KotlinVersion.KOTLIN_2_0)
}
a

Andrew Louis

02/23/2023, 12:19 PM
Got 1 min 30 on 1.8.20-Beta without K2 https://scans.gradle.com/s/aolrqhqgnsydm -- which was comparable to our fastest clean build so far However, I couldn't get a build to succeed with K2 enabled sadly; I have some jooq generated code that's resulting in some compilation failures; For some reason a java class does not fit as the argument of type
java.lang.Class<T>
:
Copy code
OrgSettings.kt:82:226 Argument type mismatch: actual type is java/lang/Class<java/lang/Module> but java/lang/Class<ft<U & Any, U?>>! was expected
t

tapchicoma

02/23/2023, 1:47 PM
@Andrew Louis could you open a Kotlin issue for this K2 problem?
a

Andrew Louis

02/23/2023, 2:09 PM
Yessir will do!
31 Views