https://kotlinlang.org logo
Join Slack
Powered by
hi everyone, I just added another multiplatform li...
# multiplatform
t
hi everyone, I just added another multiplatform library to my KMP library but when I build on macOS/Apple Silicon (targeting Android/arm64 emulator), I get this build error:
Copy code
<in Gradle task :android:mergeDebugNativeLibs>

Cause: multik-openblas-jvm-0.2.3 extracted from path ~/.gradle/caches/8.9/transforms/4d3515c442b573850b7bc131f7841f9c/transformed/multik-openblas-jvm-0.2.3/linuxX64/libmultik_jni-linuxX64.so is not an ABI
That file (along with an 
arm64
equivalent) must be shipped in that transitive dependency but why is this build even loading it? I don't have any explicit Linux targets (though I do have native targets for iOS). Everything built fine before adding these couple of new libraries (
multik-core
and 
multi-default
). More details, including my 
build.gradle.kts
on this issue. Any ideas? Thanks!
e
looks like multik-default depends on multik-openblas on JVM. neither provide any Android artifacts, but the build system will use the JVM ones if Android doesn't exist.
try multik-kotlin instead
t
Yeah, it makes sense that the Android build pulls in multik-openblas because multik-default pulls in -openblas (where available, since it's optimized) and -kotlin where it isn't. I believe I tried multik-kotlin and it didn't work immediately though I think it's because it defines some different symbols so I'll have to adjust the code as well.
It's not an ideal long-term solution since it'll mean missing out on the optimized functions but could be okay until I hear back from the upstream project
I'm still not sure why it's trying to pull in that linuxX64 library - I guess it's building for all native platforms at once? Is it possible to skip a specific target OS for native builds? I only want to target Android emulator, Android devices, iOS simulator, and iOS devices for now
e
generally JVM libraries do not have platform variants. for JNI they will either require them externally, or bundle all of them
Android cannot use desktop native libraries
t
Thanks for your help. I'll try multik-kotlin as a stop-gap and see what upstream suggests
I'm still scratching my head on this one. If I use my original config, I can build and run on iOS but hit the build error above on Android. If I replace 
multik-default
with 
multik-kotlin
(and make a small adjustment to the code to fix the build), I can build on both platforms but performance is noticeably slower on Android (emulator and device) than my previous linear algebra library so I'd really like to get the hardware acceleration that `multik-default`/`-openblas` should provide.
@Pavel Gorgulov you might have a suggestion here. Thanks!
e
multik-openblas would need to be built specifically for Android
t
@Pavel Gorgulov ah, I guess I misread the table in the 
README
- the combination JVM x 
multik-openblas
x 
androidArm64
has a ✅
does that mean something else?
it says "JVM target 
multik-openblas
for Android only supports arm64-v8a processors."
e
hmm good question. I don't see where in the build they make that work.
oh it's behind some cmake stuff
t
I just confirmed the Android emulator I'm building against is emulating an 
arm64-v8a
e
you're building an android app, it doesn't care what the target platform is until runtime
may be an AGP issue then, if the Android JNI library is present but it's failing due to the presence of other native libraries
t
it seems most likely that the issue is my build seems to be triggering a build for 
linuxX64
for some reason. I did note my config has this:
Copy code
androidTarget {
        publishAllLibraryVariants()
    }
I'm not sure if that could be related
e
no I don't think that is it. that's definitely an android-specific task that's failing
t
(either way, I should probably double-check whether I need that. I set up a local Maven repo once a long time ago but haven't needed it since and that seems related)
e
you can try
Copy code
android {
  packagingOptions {
    exclude("lib/linuxX64/libmultik_jni-linuxX64.so")
    exclude("lib/macosArm64/libmultik_jni-macosArm64.dylib")
    exclude("lib/macosX64/libmultik_jni-macosX64.dylib")
    exclude("lib/mingwX64/libmultik_jni-mingwX64.dll")
  }
}
t
that's at the top level in my KMP library's 
build.gradle.kts
, right? I still get the original error:
Copy code
:android:mergeDebugNativeLibs

Cause: multik-openblas-jvm-0.2.3 extracted from path ~/.gradle/caches/8.9/transforms/aa9c19370bfe38aa092acebe2995ebfb/transformed/multik-openblas-jvm-0.2.3/linuxX64/libmultik_jni-linuxX64.so is not an ABI
for those exclusion paths, are you basing that on the content of the depdendency package? If so, where do you see that?
e
it may need to go in the app build, not your library build
t
same issue
a setup with 
multik-default
(which pulls in 
multik-openblas
) for Android seems to be a commonly-intended use case but I haven't found any full working example 
build.gradle.kts
for that
it also seems strange that my 
~/.gradle/caches/8.9/transforms/aa9c19370bfe38aa092acebe2995ebfb/transformed/multik-openblas-jvm-0.2.3/
contains 
arm64-v8a/libmultik_jni-androidArm64.so
and 
linuxX64/libmultik_jni-linuxX64.so
but none of the others listed in the upstream config. Maybe they get filtered out earlier?
if I manually remove 
~/.gradle/caches/8.9/transforms/aa9c19370bfe38aa092acebe2995ebfb/transformed/multik-openblas-jvm-0.2.3/linuxX64/
I can build and performance looks good 😄 So I just need a fix that will ensure that even if I do a clean build
in Android studio, if I do: • a clean build (on its own), or • Invalidate Caches > Clear filesystem cache I still don't get this build error after deleting that 
linuxX64
directory But if I delete all of 
~/.gradle
, that file does get restored and the build fails as it did originally
@Pavel Gorgulov if you have ideas to solve this last problem, I'd greatly appreciate it ^ My KMP library config is basically:
Copy code
kotlin {
    ...
    sourceSets {
        commonMain.dependencies {
            ...
            implementation(libs.multik.core)
            implementation(libs.multik.default)
            implementation(libs.kotlinx.serialization.json)
        }

        commonTest.dependencies {
            implementation(libs.kotlin.test)
        }
    }
}
I just realized this is the config I tried first 🙂 😕 🙂
p
Sorry for the delayed response Using 
multik-openblas
with Android isn’t straightforward. That’s why I recommend using 
multik-kotlin
instead. Here’s some details: • You need to build LAPACK and OpenBLAS specifically for Android. I’ve been building them using GCC and GFortran, but since Google switched to Clang with SDK 20+, I had to manually configure the NDK with the necessary tools: https://github.com/devcrocod/android-gfortran. Unfortunately, I only managed to get it working on older Android versions. To fix this, the build needs to be moved entirely to Clang and Flang, but OpenBLAS has some issues in this setup: https://github.com/Kotlin/multik/issues/111. • I don’t have a dedicated artifact for Android. Native libraries are bundled together, and Android tries to preload all of them, which causes problems. At the very least, an Android-specific target is required, ideally with additional separation by architectures. • Some methods require linking with an additional library: https://github.com/Kotlin/multik/issues/127. So, here are your options: 1. Use 
multik-kotlin
only. The performance difference on mobile devices doesn’t seem significant 2. If you still want to use OpenBLAS, you need to remove all native libraries except the Android one and ensure it stays in the correct folder. It seems the solution you received should work. Try building the apk and check what’s inside it
t
@Pavel Gorgulov thanks for the detailed response! It seems like my workaround works at least in debug testing though I can see where it's bundling libraries from other platforms. I built the release 
.apk
and these are included:
Copy code
lib//macosX64
lib//macosX64/libmultik_jni-macosX64.dylib
lib//mingwX64
lib//mingwX64/libmultik_jni-mingwX64.dll
lib//x86
lib//x86/libandroidx.graphics.path.so
lib//arm64-v8a
lib//arm64-v8a/libmultik_jni-androidArm64.so
lib//arm64-v8a/libandroidx.graphics.path.so
lib//macosArm64
lib//macosArm64/libmultik_jni-macosArm64.dylib
lib//x86_64
lib//x86_64/libandroidx.graphics.path.so
Obviously, I don't want to include any platform but 
arm64-v8a
. Interesting that I'm apparently including 
libandroidx
for a couple bogus platforms as well. Maybe this is an advantage of App bundles? Do they automatically do that?
at any rate, this seems to be a related problem: if I try to build the app bundle (as I normally do), I get this error:
Copy code
Execution failed for task ':android:packageReleaseBundle'.
> A failure occurred while executing com.android.build.gradle.internal.tasks.PackageBundleTask$BundleToolWorkAction
   > File 'root/lib/macosArm64/libmultik_jni-macosArm64.dylib' uses reserved file or directory name 'lib'.
and the build fails
maybe if I could manually list in my build config the files to delete before the final 
:packageReleaseBundle
step, it would work well enough. If you know how to do that, I'd appreciate any tips
I can try switching back to 
multik-kotlin
and test on-device to see if performance is adequate. My previous solution was similar in Java (not taking advantage of the hardware) so performance should be comparable. Though, at least in the Android emulator, performance seemed worse. If it's OK on a real device, that's good enough though. And I saw some places in my own code above 
multik
that I might be able to optimize a little too
with a very brief test, release build performance seems fine on both device and emulator. Maybe debug in emulator is marginally worse but that's fine. And it's also possible I'm just imagining that
p
I built the release .apk and these are included:
As far as I know, all unnecessary libraries should also be removed, specifically 
dylib
and 
dll
. When I tested it, I encountered errors related to them, but I assume it depends on the Android version. Honestly, I’m not sure about libandroidx. it didn’t appear in my test project.
uses reserved file or directory name ‘lib’.
Yes, exactly. Even though the documentation says that libraries should be split into system-specific folders, Android still tries to preload everything from the lib folder.
Though, at least in the Android emulator, performance seemed worse.
Unfortunately, the emulator doesn’t provide a reliable performance answer since there are many nuances. It’s always better to rely on a physical device. However, I measured performance only on desktop chips, and I don’t have much knowledge about applying benchmarks on mobile devices. The context of usage is also important here: how large the arrays are, what kind of arrays they are, and how frequently various operations are applied to them.
maybe if I could manually list in my build config the files to delete before the final 
:packageReleaseBundle
step, it would work well enough. If you know how to do that, I'd appreciate any tips
You can exclude native library via `packagingOptions`:
Copy code
android {
    packagingOptions {
        exclude("lib/**/libmultik_jni-macosX64.dylib")
        exclude("lib/**/libmultik_jni-macosArm64.dylib")
        exclude("lib/**/libmultik_jni-mingwX64.dll")
        exclude("lib/**/libmultik_jni-windowsX86.dll")
        exclude("lib/macosX64/**")
        exclude("lib/macosArm64/**")
        exclude("lib/mingwX64/**")
    }
}
or
Copy code
tasks.register<Delete>("cleanNativeLibs") {
    delete(fileTree("build/intermediates") {
        include("**/*.dylib", "**/*.dll")
    })
    delete(fileTree("src/main/jniLibs") {
        include("macosX64/**", "macosArm64/**", "mingwX64/**")
    })
}

tasks.named("packageReleaseBundle") {
    dependsOn("cleanNativeLibs")
}
Actually, I'm not sure about x86 and x86_64, but you can check firstly this approach
t
@Pavel Gorgulov hmm, I couldn't get those exclusion rules to work. They should both be within my Android module 
build.gradle.kts
, right? Should the 
tasks.*
rules go within 
android {}
or at the top level? Both failed for me with 
Task with name 'packageReleaseBundle' not found in project ':android'
. When I check my gradle task list, I see 
packageReleaseBundle
but it's under 
[project] > android > Tasks > other
. Should that difference matter?
p
I’m not much of an expert on Gradle, especially in Android projects🙂 It seems like, yes, your task should be in the corresponding subproject
t
yeah, I'm certainly hitting my limits here on my Gradle skills 🙂
Unfortunately, the emulator doesn’t provide a reliable performance answer since there are many nuances. It’s always better to rely on a physical device.
definitely true. I've seen cases (but not all cases) where the Android emulator (on Apple Silicon) can perform better than an actual Android device. My main observation was that performance was noticeably worse compared to my previous linear algebra library but of course 1) I could be misremembering the prior performance as better than it was 2) I may have upgraded Android Studio and dependencies in the meantime in a way that hurt performance overall
at any rate, I think, at peak, I'm calling 
solve()
on about a hundred 63x3 matricies (along with about a hundred 
transpose()
, cross products, etc). I could believe hardware acceleration could make a visible impact but I could also believe the Kotlin implementation is good enough for my needs. But I'll also be scaling that up to potentially low-thousands of operations instead of a hundred of each
I'll probably benchmark on real hardware at some point to compare
58 Views
Open in Slack
PreviousNext
Powered by