Kotlin Native performance is poor, the following a...

# kotlin-native

[Gavin Novate]

Kotlin Native performance is poor, the following are the results tested on Android and iOS using the KMM platform, is this as expected?

val time = measureTime {
        repeat(10000) {
            floatArray = floatArrayOf(
                1f, 0f, 0f, 0f,
                0f, 1f, 0f, 0f,
                0f, 0f, 1f, 0f,
                0f, 0f, 0f, 1f
            )
        }
    }
    println("timeTest = $time")

Logs

# Android
timeTest = 1.169792ms
timeTest = 993.229us
timeTest = 978.125us
timeTest = 843.229us
timeTest = 933.854us
timeTest = 769.271us
timeTest = 1.307812ms
timeTest = 859.375us
timeTest = 938.02us
timeTest = 900us
timeTest = 1.047396ms
timeTest = 844.271us
timeTest = 1.460417ms

# iOS (iPhone 13 Pro)
timeTest = 17.078459ms
timeTest = 11.881083ms
timeTest = 14.285167ms
timeTest = 14.382958ms
timeTest = 14.487708ms
timeTest = 9.521209ms
timeTest = 17.071583ms
timeTest = 17.293083ms
timeTest = 18.362084ms
timeTest = 17.979125ms

[Christian Würthenr]

Are you testing debug or release mode?

[Gavin Novate]

I tested debug and release modes and they perform about the same.

[Christian Würthenr]

I just put your code in a unit test, so debug build and Android will be run on local machine as well:

iOS:
timeTest = 1.231667ms
timeTest = 1.391209ms
timeTest = 1.222ms
timeTest = 1.277708ms
timeTest = 1.306709ms
timeTest = 1.235334ms
timeTest = 1.208750ms
timeTest = 1.229250ms
timeTest = 1.268416ms
timeTest = 1.227041ms

Android:
timeTest = 1.535375ms
timeTest = 1.315416ms
timeTest = 1.272292ms
timeTest = 1.287792ms
timeTest = 1.258583ms
timeTest = 1.312541ms
timeTest = 799.416us
timeTest = 70.208us
timeTest = 102.584us
timeTest = 85.459us

[Christian Würthenr]

not sure if you ran it on real devices, but in that case the device itself can have an impact of course, right?

[Christian Würthenr]

Also Kotlin version might be a difference. My project is still on 1.7.10

[Gavin Novate]

Yes, I'm running on real Android (a normal mid to high end device) and iOS (iPhone 13 Pro) device.

I know it might be unscientific to compare the performance of code running on different devices.

I am trying the Compose-jb project, because it is lagging on iOS, I located the floatArrayOf method in the constructor of the Matrix class, and this method takes more than 10 times the time on iOS than on Android. I think this is Might be one of the reasons for the performance issue.

I heard from the Compose-jb team that they are also in communication with the Kotlin/Native team to resolve Compose performance issues on iOS.

[Gavin Novate]

My project is still on 1.7.10, and use experimental memory model.

[Christian Würthenr]

I have the exact same, 1.7.10 and experimental mm. I can try running it on a iPhone 12, maybe mine is as slow as yours

[Christian Würthenr]

What I did notice is that decoding JPEG images in KMM using UIImage from NSData is 10x slower than BitmapFactory on Android. Kinda unrelated, but maybe not

[Gavin Novate]

It would be great if you could test it out on the iPhone 12.

[Christian Würthenr]

timeTest = 1.743333ms
timeTest = 2.251459ms
timeTest = 1.817167ms
timeTest = 1.833542ms
timeTest = 1.855709ms
timeTest = 1.768792ms
timeTest = 1.759875ms
timeTest = 1.737041ms
timeTest = 1.778333ms
timeTest = 1.778917ms

[Christian Würthenr]

iPhone 12 mini BUT 😄 This is another project from the one posted before, this one is Kotlin 1.7.20

[Christian Würthenr]

Pixel 4a:

timeTest = 12.214688ms
timeTest = 6.964688ms
timeTest = 7.096407ms
timeTest = 6.968386ms
timeTest = 5.446668ms
timeTest = 2.506823ms
timeTest = 2.598126ms
timeTest = 2.347865ms
timeTest = 2.032917ms
timeTest = 1.995937ms

[Christian Würthenr]

Interesting. If run the test in the 1.7.20 project as a unit test I get the following:

iOS:
timeTest = 703.959us
timeTest = 849.625us
timeTest = 724.333us
timeTest = 775.792us
timeTest = 698us
timeTest = 701.917us
timeTest = 667.666us
timeTest = 667.25us
timeTest = 682.75us
timeTest = 750.625us

Android:
timeTest = 1.546459ms
timeTest = 1.377750ms
timeTest = 1.353542ms
timeTest = 15.903333ms
timeTest = 1.299458ms
timeTest = 1.300958ms
timeTest = 673.917us
timeTest = 66.291us
timeTest = 64.333us
timeTest = 70.208us

[Christian Würthenr]

The iOS times basically halfed from 1.7.10 to 1.7.20 in the unit test scenario

[Christian Würthenr]

Guess…time to update Kotlin? 😄

[Gavin Novate]

Thanks for your test, I also try to upgrade the Kotlin version.

[Christian Würthenr]

Let me know how that goes, it’s a interesting case

[Landry Norris]

It’s interesting that the times decreased in 1.7.20. I did some benchmarking previously, and found that my fibonacci test became slower on 1.7.20

[Landry Norris]

I wonder if it’s a situational thing, though. I’m working on a KMM project right now, and we see significantly better iOS performance than Android performance, to the point we decided to disable a performance operation I made on iOS, just keeping it on Android (the change led to a slight behavioural change that we didn’t want unless it was needed).

[Gavin Novate]

Sorry, this was my mistake, I consulted with my iOS colleagues and learned that I need to set the Release build in XCode, previously I had only set the Release build in Android Studio.

Now it's much better than before:

iOS
timeTest = 1.882959ms
timeTest = 1.454333ms
timeTest = 1.993334ms
timeTest = 2.583708ms
timeTest = 1.976ms
timeTest = 1.846583ms
timeTest = 825.625us
timeTest = 1.918959ms
timeTest = 1.926125ms
timeTest = 1.398083ms
timeTest = 2.054417ms
timeTest = 1.901958ms
timeTest = 2.176875ms
timeTest = 1.598959ms
timeTest = 4.244792ms
timeTest = 1.959125ms
timeTest = 1.867334ms
timeTest = 1.820625ms
timeTest = 1.934541ms
timeTest = 1.959083ms
timeTest = 1.935417ms
timeTest = 2.038500ms
timeTest = 4.428375ms
timeTest = 4.337333ms
timeTest = 2.037ms
timeTest = 1.907625ms
timeTest = 1.960500ms
timeTest = 2.016917ms
timeTest = 1.857375ms
timeTest = 2ms
timeTest = 1.945709ms
timeTest = 2.275917ms
timeTest = 2.104959ms
timeTest = 1.858500ms

[Landry Norris]

Keep in mind that you’re running a loop here. 10,000 runs ought to be enough to trigger the C2 on Android, which is very good at optimizing loops.

[Christian Würthenr]

You already see this in the numbers I posted:

timeTest = 1.546459ms
timeTest = 1.377750ms
timeTest = 1.353542ms
timeTest = 15.903333ms
timeTest = 1.299458ms
timeTest = 1.300958ms
timeTest = 673.917us
timeTest = 66.291us
timeTest = 64.333us
timeTest = 70.208us

Notice how it goes from 1.5ms to basically 0

[Landry Norris]

I wonder if that 15 is a GC run.

[Christian Würthenr]

Might be, it’s an outlier for sure

[Landry Norris]

The C2 is a really cool technology. When I first heard about the K2 compiler, I initially thought it would be a Kotlin version for K/N. Was kind of disappointed that it was a compiler frontend change.

[Christian Würthenr]

@Gavin Novate Just interesting that I got basically the same result in debug mode as my iPhone 12 run was not a release build

[Christian Würthenr]

@Landry Norris Maybe in the future 😄 But as it stands I’m already super impressed with iOS performance coming from an Android background. I implemented a MJPEG stream on Android and it took me forever to get the memory management and performance just right….on iOS it “just works”

[Landry Norris]

Also, keep in mind that JetBrains doesn’t yet make any promises on performance. The decisions they made when designing the language tend towards better performance, but for now, make sure to test it in release mode to see if it’s performant enough for your needs.

[Christian Würthenr]

If you have some critical path, e.g. some algorithm, that really needs to be fast…Kotlin interface, Swift implementation and then inject at runtime 🙂

[Christian Würthenr]

That’s the worst case from my point of view 😄

[Landry Norris]

Is Swift faster than Kotlin? There was a study that compared performance of different languages. When they normalized the performance, Swift had a score of 4.2 (1.0 is best). A separate study by JetBrains (always assume bias, though for first-party benchmarks) showed that Kotlin/Native has a score of about 2 relative to C. This makes sense when we compare to Java with its 1.9.
https://haslab.github.io/SAFER/scp21.pdf

[Christian Würthenr]

I’d just trust that Apple’s own compiler is much more optimized, but I did not do any benchmarks or something. That would just be the “worst” case if for whatever reason KMM would be too slow for something

[Landry Norris]

I believe Swift also uses LLVM, which is an Apple technology, so optimizations should mostly be upstreamed. It would be ideal if Kotlin stayed more up to date with LLVM.

[Sebastian Sellmair [JB]]

I am not an expert in benchmarking, but what makes you believe, that this was a good example?
repeat() { floatArrayOf() }

[Sebastian Sellmair [JB]]

Maybe you like this resource:
https://www.oracle.com/technical-resources/articles/java/architect-benchmarking.html ?

(Still note: Not at all an expert)

[Sebastian Sellmair [JB]]

A more practical anecdote about K/N performance: I often have to clean the build intellij.git in my daily work. I was not satisfied by the performance and verboseness of rm -rf and wrote my own tool with K/N and coroutines. It outperformed rm -rf on my machine.

[Sebastian Sellmair [JB]]

This is what I shared internally with my team:

[Sebastian Sellmair [JB]]

This was with Kotlin 1.7. I did not re-run the experiment with 1.8

[Gavin Novate]

@Sebastian Sellmair [JB]

Hi, I'm not an expert in performance testing either.

I'm trying out the Compose-jb project and it's lagging on iOS. I tried to find the cause of the lag. I found that the construction of the Matrix class is called very frequently, and its constructor uses the floatArrayOf method, so I tested the time consumption of this method.

In my initial testing, this method took longer than expected on iOS, so I raised this question.

[Nikita Lipsky]

I'm trying out the Compose-jb project and it's lagging on iOS.

I need to set the Release build in XCode.

Now it's much better than before:

@Gavin Novate, so do you still notice lagging on iOS for your compose-jb project after switch to the Release build and 1.7.20?

I heard from the Compose-jb team that they are also in communication with the Kotlin/Native team to resolve Compose performance issues on iOS

Currently, we do not have compose-jb examples with obvious lagging on iOS, so if could provide such an example we would be happy to analyze it.

[Tadeas Kriz]

The Droidcon app (https://github.com/touchlab/DroidconKotlin) is a good example of Compose UI lagging on iOS with Kotlin 1.7.20. Although it's possible there's something else happening, it doesn't lag on Android (Compose UI) or when running in SwiftUI mode on iOS.

[Nikita Lipsky]

@Tadeas Kriz -- thank you for pointing it! I see obvious lag at the sponsors tab for sponsors icons loading. I've not noticed lags at the schedule tab at first glance, but looking more closely I can agree that the scrolling is not as smooth as for Swift UI version. Do you see any other lags?

[Tadeas Kriz]

So the lagging for sponsors icon loading is probably just waiting for them to download, that's fine (also my Image implementation probably sucks so that'd be another slowing down). The main "lagging" is in the scrolling behavior. You can feel it on a 60Hz display, but on an iPhone with a 120Hz display, it's very noticeable

[Tadeas Kriz]

The whole Compose UI on iOS feels very "heavy" for lack of a better description when compared to SwiftUI