https://kotlinlang.org
Join Slack
Kotlin Native performance is poor, the following are the results tested on Android and iOS using the...
# kotlin-native
g
Kotlin Native performance is poor, the following are the results tested on Android and iOS using the KMM platform, is this as expected?
Copy code
val time = measureTime {
        repeat(10000) {
            floatArray = floatArrayOf(
                1f, 0f, 0f, 0f,
                0f, 1f, 0f, 0f,
                0f, 0f, 1f, 0f,
                0f, 0f, 0f, 1f
            )
        }
    }
    println("timeTest = $time")
Logs
Copy code
# Android
timeTest = 1.169792ms
timeTest = 993.229us
timeTest = 978.125us
timeTest = 843.229us
timeTest = 933.854us
timeTest = 769.271us
timeTest = 1.307812ms
timeTest = 859.375us
timeTest = 938.02us
timeTest = 900us
timeTest = 1.047396ms
timeTest = 844.271us
timeTest = 1.460417ms

# iOS (iPhone 13 Pro)
timeTest = 17.078459ms
timeTest = 11.881083ms
timeTest = 14.285167ms
timeTest = 14.382958ms
timeTest = 14.487708ms
timeTest = 9.521209ms
timeTest = 17.071583ms
timeTest = 17.293083ms
timeTest = 18.362084ms
timeTest = 17.979125ms
😮 1
c
Are you testing debug or release mode?
g
I tested debug and release modes and they perform about the same.
c
I just put your code in a unit test, so debug build and Android will be run on local machine as well:
Copy code
iOS:
timeTest = 1.231667ms
timeTest = 1.391209ms
timeTest = 1.222ms
timeTest = 1.277708ms
timeTest = 1.306709ms
timeTest = 1.235334ms
timeTest = 1.208750ms
timeTest = 1.229250ms
timeTest = 1.268416ms
timeTest = 1.227041ms

Android:
timeTest = 1.535375ms
timeTest = 1.315416ms
timeTest = 1.272292ms
timeTest = 1.287792ms
timeTest = 1.258583ms
timeTest = 1.312541ms
timeTest = 799.416us
timeTest = 70.208us
timeTest = 102.584us
timeTest = 85.459us
not sure if you ran it on real devices, but in that case the device itself can have an impact of course, right?
Also Kotlin version might be a difference. My project is still on 1.7.10
g
Yes, I'm running on real Android (a normal mid to high end device) and iOS (iPhone 13 Pro) device. I know it might be unscientific to compare the performance of code running on different devices. I am trying the Compose-jb project, because it is lagging on iOS, I located the floatArrayOf method in the constructor of the Matrix class, and this method takes more than 10 times the time on iOS than on Android. I think this is Might be one of the reasons for the performance issue. I heard from the Compose-jb team that they are also in communication with the Kotlin/Native team to resolve Compose performance issues on iOS.
My project is still on 1.7.10, and use experimental memory model.
c
I have the exact same, 1.7.10 and experimental mm. I can try running it on a iPhone 12, maybe mine is as slow as yours
What I did notice is that decoding JPEG images in KMM using UIImage from NSData is 10x slower than BitmapFactory on Android. Kinda unrelated, but maybe not
g
It would be great if you could test it out on the iPhone 12.
c
Copy code
timeTest = 1.743333ms
timeTest = 2.251459ms
timeTest = 1.817167ms
timeTest = 1.833542ms
timeTest = 1.855709ms
timeTest = 1.768792ms
timeTest = 1.759875ms
timeTest = 1.737041ms
timeTest = 1.778333ms
timeTest = 1.778917ms
iPhone 12 mini BUT 😄 This is another project from the one posted before, this one is Kotlin 1.7.20
Pixel 4a:
Copy code
timeTest = 12.214688ms
timeTest = 6.964688ms
timeTest = 7.096407ms
timeTest = 6.968386ms
timeTest = 5.446668ms
timeTest = 2.506823ms
timeTest = 2.598126ms
timeTest = 2.347865ms
timeTest = 2.032917ms
timeTest = 1.995937ms
Interesting. If run the test in the 1.7.20 project as a unit test I get the following:
Copy code
iOS:
timeTest = 703.959us
timeTest = 849.625us
timeTest = 724.333us
timeTest = 775.792us
timeTest = 698us
timeTest = 701.917us
timeTest = 667.666us
timeTest = 667.25us
timeTest = 682.75us
timeTest = 750.625us

Android:
timeTest = 1.546459ms
timeTest = 1.377750ms
timeTest = 1.353542ms
timeTest = 15.903333ms
timeTest = 1.299458ms
timeTest = 1.300958ms
timeTest = 673.917us
timeTest = 66.291us
timeTest = 64.333us
timeTest = 70.208us
The iOS times basically halfed from 1.7.10 to 1.7.20 in the unit test scenario
Guess…time to update Kotlin? 😄
g
Thanks for your test, I also try to upgrade the Kotlin version.
c
Let me know how that goes, it’s a interesting case
l
It’s interesting that the times decreased in 1.7.20. I did some benchmarking previously, and found that my fibonacci test became slower on 1.7.20
I wonder if it’s a situational thing, though. I’m working on a KMM project right now, and we see significantly better iOS performance than Android performance, to the point we decided to disable a performance operation I made on iOS, just keeping it on Android (the change led to a slight behavioural change that we didn’t want unless it was needed).
g
Sorry, this was my mistake, I consulted with my iOS colleagues and learned that I need to set the Release build in XCode, previously I had only set the Release build in Android Studio. Now it's much better than before:
Copy code
iOS
timeTest = 1.882959ms
timeTest = 1.454333ms
timeTest = 1.993334ms
timeTest = 2.583708ms
timeTest = 1.976ms
timeTest = 1.846583ms
timeTest = 825.625us
timeTest = 1.918959ms
timeTest = 1.926125ms
timeTest = 1.398083ms
timeTest = 2.054417ms
timeTest = 1.901958ms
timeTest = 2.176875ms
timeTest = 1.598959ms
timeTest = 4.244792ms
timeTest = 1.959125ms
timeTest = 1.867334ms
timeTest = 1.820625ms
timeTest = 1.934541ms
timeTest = 1.959083ms
timeTest = 1.935417ms
timeTest = 2.038500ms
timeTest = 4.428375ms
timeTest = 4.337333ms
timeTest = 2.037ms
timeTest = 1.907625ms
timeTest = 1.960500ms
timeTest = 2.016917ms
timeTest = 1.857375ms
timeTest = 2ms
timeTest = 1.945709ms
timeTest = 2.275917ms
timeTest = 2.104959ms
timeTest = 1.858500ms
l
Keep in mind that you’re running a loop here. 10,000 runs ought to be enough to trigger the C2 on Android, which is very good at optimizing loops.
c
You already see this in the numbers I posted:
Copy code
timeTest = 1.546459ms
timeTest = 1.377750ms
timeTest = 1.353542ms
timeTest = 15.903333ms
timeTest = 1.299458ms
timeTest = 1.300958ms
timeTest = 673.917us
timeTest = 66.291us
timeTest = 64.333us
timeTest = 70.208us
Notice how it goes from 1.5ms to basically 0
l
I wonder if that 15 is a GC run.
c
Might be, it’s an outlier for sure
l
The C2 is a really cool technology. When I first heard about the K2 compiler, I initially thought it would be a Kotlin version for K/N. Was kind of disappointed that it was a compiler frontend change.
c
@Gavin Novate Just interesting that I got basically the same result in debug mode as my iPhone 12 run was not a release build
@Landry Norris Maybe in the future 😄 But as it stands I’m already super impressed with iOS performance coming from an Android background. I implemented a MJPEG stream on Android and it took me forever to get the memory management and performance just right….on iOS it “just works”
l
Also, keep in mind that JetBrains doesn’t yet make any promises on performance. The decisions they made when designing the language tend towards better performance, but for now, make sure to test it in release mode to see if it’s performant enough for your needs.
c
If you have some critical path, e.g. some algorithm, that really needs to be fast…Kotlin interface, Swift implementation and then inject at runtime 🙂
That’s the worst case from my point of view 😄
l
Is Swift faster than Kotlin? There was a study that compared performance of different languages. When they normalized the performance, Swift had a score of 4.2 (1.0 is best). A separate study by JetBrains (always assume bias, though for first-party benchmarks) showed that Kotlin/Native has a score of about 2 relative to C. This makes sense when we compare to Java with its 1.9. https://haslab.github.io/SAFER/scp21.pdf
c
I’d just trust that Apple’s own compiler is much more optimized, but I did not do any benchmarks or something. That would just be the “worst” case if for whatever reason KMM would be too slow for something
l
I believe Swift also uses LLVM, which is an Apple technology, so optimizations should mostly be upstreamed. It would be ideal if Kotlin stayed more up to date with LLVM.
s
I am not an expert in benchmarking, but what makes you believe, that this was a good example? repeat() { floatArrayOf() }
Maybe you like this resource: https://www.oracle.com/technical-resources/articles/java/architect-benchmarking.html ? (Still note: Not at all an expert)
A more practical anecdote about K/N performance: I often have to clean the build intellij.git in my daily work. I was not satisfied by the performance and verboseness of rm -rf and wrote my own tool with K/N and coroutines. It outperformed rm -rf on my machine.
This is what I shared internally with my team:
This was with Kotlin 1.7. I did not re-run the experiment with 1.8
g
@Sebastian Sellmair [JB] Hi, I'm not an expert in performance testing either. I'm trying out the Compose-jb project and it's lagging on iOS. I tried to find the cause of the lag. I found that the construction of the Matrix class is called very frequently, and its constructor uses the floatArrayOf method, so I tested the time consumption of this method. In my initial testing, this method took longer than expected on iOS, so I raised this question.
n
I'm trying out the Compose-jb project and it's lagging on iOS.
I need to set the Release build in XCode.
Now it's much better than before:
@Gavin Novate, so do you still notice lagging on iOS for your compose-jb project after switch to the Release build and 1.7.20?
I heard from the Compose-jb team that they are also in communication with the Kotlin/Native team to resolve Compose performance issues on iOS
Currently, we do not have compose-jb examples with obvious lagging on iOS, so if could provide such an example we would be happy to analyze it.
t
The Droidcon app (https://github.com/touchlab/DroidconKotlin) is a good example of Compose UI lagging on iOS with Kotlin 1.7.20. Although it's possible there's something else happening, it doesn't lag on Android (Compose UI) or when running in SwiftUI mode on iOS.
n
@Tadeas Kriz -- thank you for pointing it! I see obvious lag at the sponsors tab for sponsors icons loading. I've not noticed lags at the schedule tab at first glance, but looking more closely I can agree that the scrolling is not as smooth as for Swift UI version. Do you see any other lags?
t
So the lagging for sponsors icon loading is probably just waiting for them to download, that's fine (also my Image implementation probably sucks so that'd be another slowing down). The main "lagging" is in the scrolling behavior. You can feel it on a 60Hz display, but on an iPhone with a 120Hz display, it's very noticeable
The whole Compose UI on iOS feels very "heavy" for lack of a better description when compared to SwiftUI
151 Views
Open in Slack
PreviousNext
Powered by

kotlinlang

A modern programming language that makes developers happier.

Powered by