https://kotlinlang.org logo
Join Slack
Powered by
Anyone have any tips on how to speed up github act...
# kotlin-native
r
Anyone have any tips on how to speed up github actions when building kotlin native binaries? I am burning a lot of minutes on my macos and windows builds (particularly macos b/c of their minutes multiplier). Seems like a good chunk of the time goes into downloading konan dependencies
For reference, my latest release builds https://github.com/unredundant/neonctl/actions/runs/5046959605 Ubuntu took 3 min Macos 7 min windows 8 min
h
Downloading and extracting Konan takes only few seconds (add a BuildScan to check it). Building the native library does take ages. On macOS, you can't build native targets in parallel and there is no real workaround yet: https://youtrack.jetbrains.com/issue/KT-49385/Kotlin-Native-Parallelize-link-tasks-for-different-targets
r
anecdotally, there's no way this is true, at least on a stock github action runner. I'm watching some builds right now (macos/windows specifically) where download and extract of konan deps pushes way past the 4 minute mark
luckily, i think there is a problem where my os matrix is causing cache key collisions, so hopefully if I resolve that, the konan deps will be cached in future builds 🤞
h
The question is, is caching it faster? You still need to download and extract the cache. But yeah, it took more time than a few seconds, sorry.
Copy code
Please wait while Kotlin/Native compiler 1.8.21 is being installed.
Download <https://download.jetbrains.com/kotlin/native/builds/releases/1.8.21/macos-x86_64/kotlin-native-prebuilt-macos-x86_64-1.8.21.tar.gz> (308.22 MB)
Download kotlin-native-prebuilt-macos-x86_64-1.8.21.tar.gz finished, took 24 ms
Unpack Kotlin/Native compiler to /Users/runner/.konan/kotlin-native-prebuilt-macos-x86_64-1.8.21
Unpack Kotlin/Native compiler to /Users/runner/.konan/kotlin-native-prebuilt-macos-x86_64-1.8.21 finished, took 24 s 260 ms
https://github.com/cashapp/sqldelight/actions/runs/5041216230/jobs/9040718326#step:7:30
r
that is a question for sure 🙂
h
BTW: This is your run, 4 + 21 seconds:
Copy code
Please wait while Kotlin/Native compiler 1.8.21 is being installed.
Download <https://download.jetbrains.com/kotlin/native/builds/releases/1.8.21/macos-x86_64/kotlin-native-prebuilt-macos-x86_64-1.8.21.tar.gz> (308.22 MB)
Download kotlin-native-prebuilt-macos-x86_64-1.8.21.tar.gz finished, took 4 s 237 ms
Unpack Kotlin/Native compiler to /Users/runner/.konan/kotlin-native-prebuilt-macos-x86_64-1.8.21
Unpack Kotlin/Native compiler to /Users/runner/.konan/kotlin-native-prebuilt-macos-x86_64-1.8.21 finished, took 21 s 851 ms
r
Yes, I am calling shenanigans on that. Maybe there is additional steps that gradle is not taking into account in that number, that would require a build scan to introspect, but something is taking 4+ minutes, prior to the binary creation step
all i know presently is that it takes the github action about 5 minutes for gradle to tell me that it took 20s. I'm no math wiz but that doesn't add up 😆
a
are you using the 
gradle/gradle-build-action
? Do you have Gradle build cache enabled?
r
a
you can also add a step to cache the Kotlin Konan dir. Here’s what I use:
Copy code
- name: Cache Kotlin Konan
        id: cache-kotlin-konan
        uses: actions/cache@v3
        with:
          path: |
            ~/.konan/**/*
          key: kotlin-konan-${{ runner.os }}
r
ah yes will likely need to add that directory, good call!
h
I really recommend you to use a build scan, this makes analyzing way easier.
r
is there a way to get build scans to work in CI? every time I use a build scan gradle always sends me thru this super annoying loop of command line opt ins and email hyperlinks
h
a
there’s no way to workaround the need to click the link and give it your email, unless you pay for Gradle Enterprise
h
Just remove the jdk resolver plugin
r
ah yes I should specify, I'm a lowly solo dev in this endeavor... gradle enterprise is not going to be an option for me 🥲
a
me too :) But you can still enable the scans and then you can click the link and generate the scan result if you need to, for example if you’re asking for help on improving build speed blob wink
h
There is also a free option, but they are publicly accessible, which sounds fine with open source projects and using public GitHub actions.
r
only been one run, but fixing the cache key issue dropped my macos builds from 7 minutes to 3 and my windows build from 6 to 3
it's possible that this is before even taking the native compilation into account, this might just be that I wasn't getting any caching at all on these builds due to key conflict
a
add
Copy code
org.gradle.caching=true
to 
gradle.properties
- it gives a really big improvement
r
no i have that already, what i'm saying is that in github actions, the os matrix was causing my cache key to collide b/c i was not properly accounting for the os names
by fixing that, I cut build times in half 🙂
actually... on this repo.. i don't have that 😱
a
r
i usually start from a template
good catch!
though i'm not too familiar with what that property actually does... assume that gradle would cache by default yes? if that is not enabled, does gradle really start from scratch on every command?
Copy code
By default, the build cache is not enabled. You can enable the build cache in a couple of ways:

Run with --build-cache on the command-line
Gradle will use the build cache for this build only.

Put org.gradle.caching=true in your gradle.properties
Gradle will try to reuse outputs from previous builds for all builds, unless explicitly disabled with --no-build-cache.

When the build cache is enabled, it will store build outputs in the Gradle user home.
https://docs.gradle.org/current/userguide/build_cache.html
interesting
a
Gradle does lots of caching, and what Build Cache solves is saving & sharing that cache so that even on a fresh checkout, the cache is still available in the 
$HOME/.gradle
dir
which of course helps on CI/CD, when it’s freshly checked out every time
r
thanks for the help! @Adam S & @hfhbd
a
and in theory you could share the cache via a HTTP server or a AWS S3 bucket or something, so if I checked out your repo and ran 
./gradlew check
it would fetch the result from the build cache, see that the inputs haven’t changed, and then conclude that the tests are correct without actually running them.
r
my employer made some pretty crazy CI speed gains by doing something similar with their monolith (though in python land, not w/ gradle)
a
that’s cool
214 Views
Open in Slack
PreviousNext
Powered by