Hi guys! We've been getting an increasing amount ...

# kotlin-native

[andreasmattsson]

Hi guys!

We've been getting an increasing amount of crash reports (via Crashlytics) in our production app with Kotlin Native as a static framework, of a type we have been unable to reproduce with the debugger attached / on our own devices. All of them

EXC_BAD_ACCESS KERN_INVALID_ADDRESS

and pointing to

(anonymous namespace)::garbageCollect(MemoryState*, bool)

Any suggestions for how to troubleshoot this? It seems to be happening in pretty random places (e.g. the rest of the stack can look quite different from place to place, and analytics events suggests it is happening in a bunch of different views.

A few more (partial) sample stack traces:

Crashed: com.apple.main-thread
0  Deedster                       0x1051847e4 (anonymous namespace)::garbageCollect(MemoryState*, bool) + 4313057252
1  Deedster                       0x105194b40 ObjHeader* (anonymous namespace)::allocInstance<true>(TypeInfo const*, ObjHeader**) + 4313123648
2  Deedster                       0x104a7fbd0 kfun:kotlin.collections.HashMap.hash#internal + 4305697744
3  CoreFoundation                 0x193fa9938 -[NSDictionary getObjects:andKeys:count:] + 240
4  CoreFoundation                 0x194114064 __NSDictionaryEnumerate + 624
5  Foundation                     0x194530f50 _writeJSONObject + 424
6  Foundation                     0x194531d1c ___writeJSONArray_block_invoke + 268
7  CoreFoundation                 0x19409995c __NSARRAY_IS_CALLING_OUT_TO_A_BLOCK__ + 16

0   Deedster                       0x100a507e4 (anonymous namespace)::garbageCollect(MemoryState*, bool) + 4381165540
1   Deedster                       0x100883658 objc2kotlin.3124 + 4379276888
2   ReactiveSwift                  0x102ed9068 $s13ReactiveSwift6SignalC2on5event6failed9completed11interrupted10terminated8disposed5valueACyxq_GyAC5EventOyxq__GcSg_yq_cSgyycSgA3RyxcSgtFyAC8ObserverCyxq__G_AA8LifetimeCtXEfU_yAOcfU_ + 488
3   ReactiveSwift                  0x102eea9a4 $s13ReactiveSwift6SignalC2on5event6failed9completed11interrupted10terminated8disposed5valueACyxq_GyAC5EventOyxq__GcSg_yq_cSgyycSgA3RyxcSgtFyAC8ObserverCyxq__G_AA8LifetimeCtXEfU_yAOcfU_TA + 52
4   ReactiveSwift                  0x102ebfc28 $s13ReactiveSwift6SignalC8ObserverC4sendyyAC5EventOyxq__GF + 40
5   ReactiveSwift                  0x102ed3604 $s13ReactiveSwift6SignalC4Core33_6DF632AE8A9288C3EAD8EFDF3D3AF99ELLC4sendyyAC5EventOyxq__GF + 780
6   ReactiveSwift                  0x102ebfc28 $s13ReactiveSwift6SignalC8ObserverC4sendyyAC5EventOyxq__GF + 40
7   ReactiveSwift                  0x102ef1354 $s13ReactiveSwift15TransformerCore33_8C2AEE826CBF26B7535B1491D314A4F4LLC5startyAA10Disposable_pAA6SignalC8ObserverCyxq__GAaF_pXEFAJyq0_q1__GAaF_pXEfU_yAH5EventOyxq__GcfU_ + 84

Crashed: com.apple.main-thread
0  Deedster                       0x1017e47e4 (anonymous namespace)::garbageCollect(MemoryState*, bool) + 4388997092
1  Deedster                       0x1017f490c ReleaseHeapRefStrict + 4389062924
2  Deedster                       0x1017f1a64 ReadHeapRefLocked + 4389050980
3  Deedster                       0x10114c7d4 kfun:kotlinx.coroutines.CancellableContinuationImpl.<get-state>$kotlinx-coroutines-core()kotlin.Any? + 4382083028
4  Flutter                        0x102fb532c (Missing)
5  Flutter                        0x102f558fc (Missing)
6  Flutter                        0x102fa68dc (Missing)
7  Flutter                        0x102f63628 (Missing)

[andreasmattsson]

We rely pretty heavily on Kotlin Coroutines in our common library, and then bridging these to `ReactiveSwift`/`ReactiveObjC` to listen to ViewModel updates on the native iOS side. Not sure if that is relevant.

[Artyom Degtyarev [JB]]

Hello! The best thing can be done here is providing a reproducer project to work with. Is there any way K/N team can have a look at the code?

[andreasmattsson]

Hmm... Unfortunately, I don't think I can share the entire code with you guys (at least that would require me getting approval with a process that might take some time).
And I'm unsure how to make a smaller reproducer project either, as we haven't actually been able to reproduce these crashes ourselves. We've only seen them reported from Crashlytics.

[svyatoslav.scherbina]

Do you have Kotlin classes extending Objective-C classes in your code?

[andreasmattsson]

Hrm... Not much, but could be a handful cases. Might take me some time to look through it all but I found this case:

class KotlinJsonMessageCodec : NSObject(), FlutterMessageCodecProtocol {

FlutterMethodCodecProtocal being imported with Cocoapods:

import cocoapods.Flutter.*

Flutter is at the moment the only cocoapod we use. It's been a while since I wrote that code, but I'm pretty sure I was forced somehow to subclass NSObject in order to also implement the FlutterMessageCodecProtocol.

[andreasmattsson]

The reverse is much more common.
(interfaces defined in Kotlin common, implemented by Swift classes, and then called from the common module).

[andreasmattsson]

Did some more searching in the codebase and I think actually the only cases are protocols from the Flutter Cocoapod that are implemented by Kotlin classes that also subclass NSObject.

[andreasmattsson]

Trying to see what the stacks of the crashes have in common... Most of them at some point or other pass through a dispatcher to ensure the code is run the iOS UI/Main loop. Do you guyes see any potential problems with this implementation:

import kotlinx.coroutines.*
import platform.darwin.*
import platform.Foundation.NSThread
import kotlin.coroutines.CoroutineContext

@UseExperimental(InternalCoroutinesApi::class)
private object MainLoopDispatcher : CoroutineDispatcher(), Delay {

    override fun dispatch(context: CoroutineContext, block: Runnable) {
        if (NSThread.isMainThread()) {
            block.run()
        } else {
            dispatch_async(dispatch_get_main_queue()) {
                block.run()
            }
        }
    }

    @UseExperimental(ExperimentalCoroutinesApi::class)
    @InternalCoroutinesApi
    override fun scheduleResumeAfterDelay(timeMillis: Long, continuation: CancellableContinuation<Unit>) {
        dispatch_after(dispatch_time(DISPATCH_TIME_NOW, timeMillis * 1_000_000), dispatch_get_main_queue()) {
            with(continuation) {
                resumeUndispatched(Unit)
            }

        }
    }

    @InternalCoroutinesApi
    override fun invokeOnTimeout(timeMillis: Long, block: Runnable): DisposableHandle {
        val handle = object : DisposableHandle {
            var disposed = false
                private set

            override fun dispose() {
                disposed = true
            }
        }
        dispatch_after(dispatch_time(DISPATCH_TIME_NOW, timeMillis * 1_000_000), dispatch_get_main_queue()) {
            if (!handle.disposed) {
                block.run()
            }
        }

        return handle
    }
}

internal actual val Dispatchers.MainLoop: CoroutineDispatcher get() = MainLoopDispatcher

internal actual val Dispatchers.Flow: CoroutineDispatcher get() = MainLoop

Adapted from here:
https://github.com/Kotlin/kotlinx.coroutines/issues/470#issuecomment-440080970

[svyatoslav.scherbina]

Did some more searching in the codebase and I think actually the only cases are protocols from the Flutter Cocoapod that are implemented by Kotlin classes that also subclass NSObject.

Can references to these objects potentially leak to other threads?

[andreasmattsson]

I doubt it, the Flutter docs specify that events from Flutter platform channels will be triggered on the main thread, and that events in the other direction need to be called from the main thread, so I've taken this into account.
I don't think I can say with 100% certainty that they can't leak to other threads though... Would it count as leaking to another thread as only storing a reference to one of these objects on another thread, even though no methods are ever called on it from anyplace else than the main thread? (not saying that happens, just that my confidence that mere references don't exist on other threads is lesser)

[andreasmattsson]

Would that be the kind of thing that could cause these kinds of crashes?

[svyatoslav.scherbina]

I don’t think I can say with 100% certainty that they can’t leak to other threads though... Would it count as leaking to another thread as only storing a reference to one of these objects on another thread, even though no methods are ever called on it from anyplace else than the main thread?

Unfortunately, it would count currently.

Would that be the kind of thing that could cause these kinds of crashes?

This is possible.

Have you successfully reproduced the crashes on your side?

[andreasmattsson]

Unfortunately still haven't been able to reproduce them on our side. Not sure if that is because it only happens in release builds or if it's just rare enough that it happens.

[svyatoslav.scherbina]

Could you please try to reproduce the crash locally with release build?

[andreasmattsson]

We'll certainly try and I will report back if we manage. Unfortunately (blessing and a curse I guess) the crashes don't seem to happen often enough that I can dedicate 100% of my time to this at the moment.

[svyatoslav.scherbina]

Ok, that’s perfectly understandable.
Thank you anyway!

[andreasmattsson]

Since these objects are passed over to a third-party framework (Flutter) it's difficult for me to guarantee they don't end up being referenced in other threads. Do you think we could insulate our Kotin objects by wrapping them in Swift objects that store the references to the Kotlin objects in something like Thread.current.threadDictionary ?

[svyatoslav.scherbina]

Do you think we could insulate our Kotin objects by wrapping them in Swift objects that store the references to the Kotlin objects

This would be enough, even without Thread.current.threadDictionary part.

[andreasmattsson]

Ah, I see. 🙂 Thanks, I'll see what I can do.
We have few enough classes that subclass NSObject (or other objC classes) that refactoring it shouldn't take too long.

[andreasmattsson]

@svyatoslav.scherbina Hi!
I've been working on refactoring so that no Kotlin objects subclass NSObject anymore, and instead have the protocols required by Flutter implemented on the Swift side and delegating to Kotlin objects.

I just noticed however that there are methods from the Flutter SDK where I need to pass these protocol implementations that require them to also be NSObject instances. Can I safely subclass NSObject in my Swift classes that store references to kotlin classes, or is that just as bad/likely to cause these memory issues as if the kotlin objects subclassed NSObject directly?

[svyatoslav.scherbina]

Hi.

Can I safely subclass NSObject in my Swift classes that store references to kotlin classes

Yes, this is totally ok.