I've updated my Kotlin Native project to use K2, a...

# kotlin-native

[Adam S]

I've updated my Kotlin Native project to use K2, and I'm getting an unexpected 'concurrent modification' exception. I can't see any obvious cause, but what's confusing me is that I see Kotlin/Wasm in the stacktrace.

(/opt/buildAgent/work/b2e1db4d8d903ca4/kotlin/libraries/stdlib/native-wasm/src/kotlin/collections/AbstractMutableList.kt:138:23)
    at 5   exampleAudioMixedProcessor.kexe     0x1029658cb        kfun:kotlin.collections.AbstractMutableList.IteratorImpl.next#internal + 151

Why might Kotlin Native be using Wasm?

[Adam S]

Full error:

Uncaught Kotlin exception: 
kotlin.ConcurrentModificationException
    at 0   exampleAudioMixedProcessor.kexe     0x1029348ff        kfun:kotlin.Exception#<init>(kotlin.String?;kotlin.Throwable?){} + 143 (/opt/buildAgent/work/b2e1db4d8d903ca4/kotlin/kotlin-native/runtime/src/main/kotlin/kotlin/Exceptions.kt:25:70)
    at 1   exampleAudioMixedProcessor.kexe     0x102934a97        kfun:kotlin.RuntimeException#<init>(kotlin.String?;kotlin.Throwable?){} + 143 (/opt/buildAgent/work/b2e1db4d8d903ca4/kotlin/kotlin-native/runtime/src/main/kotlin/kotlin/Exceptions.kt:36:70)
    at 2   exampleAudioMixedProcessor.kexe     0x1029356e3        kfun:kotlin.ConcurrentModificationException#<init>(kotlin.String?;kotlin.Throwable?){} + 143 (/opt/buildAgent/work/b2e1db4d8d903ca4/kotlin/kotlin-native/runtime/src/main/kotlin/kotlin/Exceptions.kt:175:65)
    at 3   exampleAudioMixedProcessor.kexe     0x102935757        kfun:kotlin.ConcurrentModificationException#<init>(){} + 95 (/opt/buildAgent/work/b2e1db4d8d903ca4/kotlin/kotlin-native/runtime/src/main/kotlin/kotlin/Exceptions.kt:178:35)
    at 4   exampleAudioMixedProcessor.kexe     0x102965ccb        kfun:kotlin.collections.AbstractMutableList.IteratorImpl.checkForComodification#internal + 211 (/opt/buildAgent/work/b2e1db4d8d903ca4/kotlin/libraries/stdlib/native-wasm/src/kotlin/collections/AbstractMutableList.kt:138:23)
    at 5   exampleAudioMixedProcessor.kexe     0x1029658cb        kfun:kotlin.collections.AbstractMutableList.IteratorImpl.next#internal + 151 (/opt/buildAgent/work/b2e1db4d8d903ca4/kotlin/libraries/stdlib/native-wasm/src/kotlin/collections/AbstractMutableList.kt:120:13)
    at 6   exampleAudioMixedProcessor.kexe     0x1029aa573        kfun:kotlin.collections.Iterator#next(){}1:0-trampoline + 99 (/opt/buildAgent/work/b2e1db4d8d903ca4/kotlin/libraries/stdlib/native-wasm/src/kotlin/collections/Iterator.kt:18:21)
    at 7   exampleAudioMixedProcessor.kexe     0x1028823f3        kfun:audio_mixed_processor#main(){} + 2843 (/[...]/examples/src/audio/audio_mixed_processor.kt:<unknown>)
    at 8   exampleAudioMixedProcessor.kexe     0x1028829a7        Konan_start + 111 (/[...]/examples/src/audio/audio_mixed_processor.kt:53:1)
    at 9   exampleAudioMixedProcessor.kexe     0x10288cafb        Init_and_run_start + 107
    at 10  dyld                                0x1831920df        start + 2359

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':documentation:examples:runExampleAudioMixedProcessorDebugExecutableMacosArm64'.
> Process 'command '/[...]/examples/build/bin/macosArm64/exampleAudioMixedProcessorDebugExecutable/exampleAudioMixedProcessor.kexe'' finished with non-zero exit value 134

[Oleg Yukhnevich]

Why might Kotlin Native be using Wasm?

It’s not, there are stdlib sources that are shared between wasm and native

AFAIR there were some changes in Kotlin 2.0 regarding improving performance and stability of collections for K/N
So there are 2 variants:
1. It’s a bug in new implementation (probably not)
2. It’s a bug in your code, because previously there were no check for mutation
Also, concurrent modification exception could be not only in case of concurrency
F.e if when iterating a list you are removing/adding item to itself

[Adam S]

ahh I see, so yeah, Native uses Wasm sources

https://github.com/JetBrains/kotlin/blob/56ba701b3d05738d70a4f906859bbb066c9531a0/libraries/stdlib/build.gradle.kts#L588-L589

[Adam S]

thanks Oleg!

[Adam S]

there might not be a bug in the list, but something has changed with Kotlin. Maybe it's much faster or much slower, or maybe something like functions that are captured by staticCFunction() now run in a background process...

[Oleg Yukhnevich]

Probably having a code where the error happens and second pair of eyes helps :)

[Adam S]

well, if you're offering to help, that'd be nice! :)

[Adam S]

Basically Raylib has a way of providing an external function to handle audio update events. I've passed in a staticCFunction that computes the average audio level, and stores the history in an ArrayDeque.

Here's my hook: https://github.com/adamko-dev/kayray/blob/10912a32c413a0760d7797ffeba321c0e392a388/documentation/examples/src/audio/audio_mixed_processor.kt#L25-L48

And here's where it's attached: https://github.com/adamko-dev/kayray/blob/2227b4448a5adb24e2bd12e0d0ae7bc9b09c1f81/kayray-modules/kayray-raylib/src/nativeMain/kotlin/scopes/AudioDevice.kt#L35

[Oleg Yukhnevich]

Thanks, I will take a look later today!
Not near PC right now

[Adam S]

I think the answer is just going to be: use a concurrent list, instead of an ArrayDeque

[Adam S]

Hmmm I am leaning towards there being some sort of subtle bug with mutable lists.

This is basically what I have now, where processor() is called by C and updates a global mutable list. It worked in 1.9.24, but after updating to 2.0.0 I get a ConcurrentModificationException

private val averages = ArrayDeque<Float>(400)

private fun processor(frames: List<Float>) {
  val average = frames.average().toFloat()
  // Moving history to the left
  while (averages.size >= 400) {
    averages.removeFirst()
  }
  // Adding last average value
  averages.addLast(average)
}

Even if I try and update the averages atomically using a MutableStateFlow, then I still get a ConcurrentModificationException

private val averages = MutableStateFlow(ArrayDeque<Float>(400))

private fun processor(frames: List<Float>) {
  val average = frames.average().toFloat()
  averages.update { value ->
    // Moving history to the left
    while (value.size >= 400) {
      value.removeFirst()
    }
    // Adding last average value
    value.addLast(average)
    value
  }
}

[Oleg Yukhnevich]

I do believe that the reason why it started to fail is this commit: https://github.com/JetBrains/kotlin/commit/c9d8ecc5996cf4409dd9d5404f857e683ce96b49
and as far as I see you have averageVolume.forEachIndexed usage below - which cause the error, as most probably callback (processAudio) can be called concurrently, right?

[Michael Paus]

This looks to me like a typical usage error of collections as described here:
https://www.baeldung.com/kotlin/concurrent-modification-exception

[Adam S]

most likely @Michael Paus, but I still saw the same issue, even when trying to guard against it, as suggested in the article. I am suspecting there's a subtle bug in ArrayDeque + the new protections. Very weird! I haven't had a chance to look at it again...