Hi <@UHAJKUSTU>, I'm investigating some test cases...

# reaktive

[Omar Mainegra]

Hi @Arkadii Ivanov, I'm investigating some test cases taking too long in iOS, and I narrowed it to concatMap .

This is what I found

@Test
fun Operator_durations() {
    val concatMapDuration = measureTime {
        (1..100).asObservable().concatMap { (1..100).asObservable() }.test().values
    }
    val flatMapDuration = measureTime {
        (1..100).asObservable().flatMap { (1..100).asObservable() }.test().values
    }

    println("Measured: concatMap=$concatMapDuration, flatMap=$flatMapDuration")
}

Output

[iOS] Measured: concatMap=7.68s, flatMap=7.47s
[JVM] Measured: concatMap=122ms, flatMap=36.6ms
[JS ] Measured: concatMap=241ms, flatMap=147ms

As you can see iOS time is a magnitude higher

[Arkadii Ivanov]

Hey, I will check, this might be related to https://youtrack.jetbrains.com/issue/KT-39160

[Arkadii Ivanov]

Would you mind to raise an issue?

[Omar Mainegra]

For sure, thanks

[Arkadii Ivanov]

Also please make sure you are testing release version

[Arkadii Ivanov]

Because debug native builds are much slower currently

[Omar Mainegra]

Ok, I'll do that too

[Omar Mainegra]

Hey, how can I run the release test? I can only seem to be able to link the debug

[Omar Mainegra]

./gradlew tasks|grep link

linkDebugFrameworkIos - Links a framework 'debugFramework' for a target 'ios'.
linkDebugFrameworkIosArm64 - Links a framework 'debugFramework' for a target 'iosArm64'.
linkDebugTestIos - Links a test executable 'debugTest' for a target 'ios'.
linkDebugTestIosArm64 - Links a test executable 'debugTest' for a target 'iosArm64'.
linkReleaseFrameworkIos - Links a framework 'releaseFramework' for a target 'ios'.
linkReleaseFrameworkIosArm64 - Links a framework 'releaseFramework' for a target 'iosArm64'.
kotlinNpmInstall - Find, download and link NPM dependencies and projects

[Arkadii Ivanov]

Hey, you can try:

macosX64().binaries.getTest("DEBUG"). optimized = true
macosX64().binaries.getTest("DEBUG").debuggable = false

But I never used this. Better to not use tests for benchmarks. Try normal release bulid.

[Arkadii Ivanov]

Also AFAIK iOS tests are executing in a simulator process, whereas others are executing directly in the host. This should also affect results.

[Omar Mainegra]

Hi there, I created a macOS target and it's like you said:
Release:
Measured: concatMap=14.1ms, flatMap=12.9ms
Debug:
Measured: concatMap=39.8ms, flatMap=35.1ms

[Omar Mainegra]

Also yes, the iOS tests are executed in the simulator, not directly in the host, but it is X64 , is not like is virtualized or anything. I really don't see any reason for such a bad performance

[Arkadii Ivanov]

So first of all I am surprised for such a good performance on macOS, it is faster than JVM??? Probably because JVM tests have to be warmed up, this affects results a lot.

[Arkadii Ivanov]

As of iOS performance, this is interesting. The Reaktive's code is same for macOS and iOS (there is little to no difference). I would still try release build on a real device. Like run the benchmark on a button click and measure the time.

[Arkadii Ivanov]

Would you mind to check this?

[Omar Mainegra]

Yes, ☝️ this is my next step

[Omar Mainegra]

BTW I run the testReleaseUnitTest and this is the result (without using TestObserver)

[Omar Mainegra]

Measured: concatMap=11.595ms, flatMap=4.355ms

[Omar Mainegra]

Is better than macOS release, even tho is impressive the performance of macOS

[Omar Mainegra]

Ok so, I run the code snippet in an iOS app running in the simulator, both DEBUG and RELEASE configurations, and the result were really closed to the macOS app, that makes me realize I wasn't use the TestObservableObserver (since it wasn't a test) but a plain subscribe. This is the new measure

[Omar Mainegra]

@Test
fun Operator_durations() {
    val N = 100

    var sum1 = 0
    var sum2 = 0
    var sum3 = 0
    val expectedSum = N*(N + 1)/2*N

    val duration1 = measureTime {
        sum1 = (1..100).asObservable().concatMap { (1..100).asObservable() }.test().values.sum()
    }

    val duration2 = measureTime {
        (1..100).asObservable().concatMap { (1..100).asObservable() }.subscribe { sum2 += it }
    }

    val duration3 = measureTime {
        val list = AtomicReference<List<Int>>(emptyList())
        (1..100).asObservable().concatMap { (1..100).asObservable() }.subscribe { list.value = list.value + it }
        sum3 = list.value.sum()
    }

    assertEquals(expectedSum, sum1)
    assertEquals(expectedSum, sum2)
    assertEquals(expectedSum, sum3)

    println("Measured: duration1=$duration1, duration2=$duration2, duration3=$duration3")
}

[Omar Mainegra]

The output is:
Measured: duration1=10.9s, duration2=43.2ms, duration3=7.39s

[Omar Mainegra]

Keeping the experiment, I run it in DEBUG and RELEASE in the iOS app, RELEASE is way faster, but still slow compared to not using AtomicReference<List<Int>>

// DEBUG
Measured: duration1=49.5ms, duration2=6.69s
Measured: duration1=69.5ms, duration2=6.57s
Measured: duration1=63.9ms, duration2=6.70s

// RELEASE
Measured: duration1=26.9ms, duration2=814ms
Measured: duration1=25.3ms, duration2=795ms
Measured: duration1=26.7ms, duration2=786ms

[Arkadii Ivanov]

Thanks for such a good investigation. This confirms that the issue is with copying the list. It is being copying 10000 times which is very painful in Kotlin Native. I reported exactly the same issue recently, so you can vote for it and track. There is even an explanation of why is it slow. https://youtrack.jetbrains.com/issue/KT-39160

[Arkadii Ivanov]

Btw, we can improve TestObservableObserver so it will use mutable list by default and fallback to immutable if frozen. This will dramatically improve performance in such cases. You will also need to call it like .test(autoFreeze = false). What do you think?

[Omar Mainegra]

That sounds great, I'm gonna try doing it, and if I have success I'll PR

[Arkadii Ivanov]

Nice! This should be simple. The onNext callback is already synchronized by definition. So just initialize the AtomicReferenec with an empty mutable list and every time check whether it is frozen or not. If not then just add the item, otherwise fallback to the current approach.

[Arkadii Ivanov]

I have submitted a PR: https://github.com/badoo/Reaktive/pull/475

[Omar Mainegra]

Looks great, many thanks