I’m wondering if anybody could give some pointers ...

# kotlin-native

[katokay]

I’m wondering if anybody could give some pointers (pun unintended) 😛 on how to resolve an issue I’m facing with cinterop in a multi-platform project I’m building? Here are some of the details.
• Project is located https://github.com/CoreyKaylor/kotlin-lmdb
• The project depends on native compiled lmdb key-value store
• There is a commonMain, jvmMain, nativeMain initially created from the IntelliJ new project dialog.
• There are tests located in commonTest that cover both jvm and native.
• jvm is using jnr-ffi package for the interop and all tests are passing at this point here
• native tests are flakey and fail roughly 80% of the time
• I believe it’s possible the cause is the initial creation of the Env class that passes a double pointer to the native mdb_env_create function
• The failure actually occurs in the mdb_env_open function, but that depends on the outcome from the create function
• That said, it passes sometimes so I’m confused. Here is a snippet of the env creation code, but again I’m not sure the problem even lies here.

import kotlinx.cinterop.*
import lmdb.*

actual class Env : AutoCloseable {
    private val envPtr: CPointerVar<MDB_env> = memScoped { allocPointerTo() }
    internal val ptr: CPointer<MDB_env>

    init {
        check(mdb_env_create(envPtr.ptr))
        ptr = envPtr.value!!
    }

    private var isOpened = false

    actual var maxDatabases: UInt = 0u
        set(value) {
            check(mdb_env_set_maxdbs(ptr, value))
            field = value
        }

    actual var mapSize: ULong = 1024UL * 1024UL * 50UL
        set(value) {
            check(mdb_env_set_mapsize(ptr, value))
            field = value
        }

    actual var maxReaders: UInt = 0u
        set(value) {
            check(mdb_env_set_maxreaders(ptr, value))
            field = value
        }

    actual val stat: Stat?
        get() {
        val statPtr: CValue<MDB_stat> = cValue<MDB_stat>()
        check(mdb_env_stat(ptr, statPtr))
        val pointed = memScoped {
            statPtr.ptr.pointed
        }
        return Stat(
            pointed.ms_branch_pages, pointed.ms_depth, pointed.ms_entries, pointed.ms_leaf_pages,
            pointed.ms_overflow_pages, pointed.ms_psize
        )
    }

    actual val info: EnvInfo?
        get() {
            val envInfo: CValue<MDB_envinfo> = cValue<MDB_envinfo>()
            check(mdb_env_info(ptr, envInfo))
            val pointed = memScoped {
                envInfo.ptr.pointed
            }
            return EnvInfo(pointed.me_last_pgno, pointed.me_last_txnid,
                pointed.me_mapaddr.toLong().toULong(), pointed.me_mapsize, pointed.me_maxreaders, pointed.me_numreaders)
        }

    actual fun open(path: String, vararg options: EnvOption, mode: UShort) {
        check(mdb_env_open(ptr, path, options.asIterable().toFlags(), mode))
        isOpened = true
    }

    actual fun beginTxn(vararg options: TxnOption) : Txn {
        return Txn(this, *options)
    }

    actual fun copyTo(path: String, compact: Boolean) {
        val flags = if (compact) {
            0x01u
        } else {
            0u
        }
        check(mdb_env_copy2(ptr, path, flags))
    }

    actual fun sync(force: Boolean) {
        val forceInt = if(force) 1 else 0
        check(mdb_env_sync(ptr, forceInt))
    }

    actual override fun close() {
        if (!isOpened)
            return
        mdb_env_close(ptr)
    }
}

[Landry Norris]

The memscoped method will free all memory allocated in the lambda when it completes. Your envPtr is technically invalid memory from the start.

[Landry Norris]

I would recommend using an Arena for allocations that have the same lifecycle as the object (with a close method that clears the Arena), or a cleaner if you feel confident with cleaners.

[Oleg Yukhnevich]

IMO, in this case arena is not needed, but as pointed upper, pointer allocated by allocPointerTo inside memScoped will be inaccessible just after exit from memScoped block
So, you just need to move memScoped to constructor (or variable declaration), f.e.:

internal val ptr: CPointer<MDB_env> = memScoped {
  val ptrVar = allocPointerTo<MDB_env>()
  check(mdb_env_create(ptrVar.ptr))
  checkNotNull(ptrVar.value)
}

and then, of course, don’t forget to free it, like you do it in close method, though, I prefer Cleaner as pointed upper
if to use Cleaner, you will need to add one more variable after ptr declaration, like:

private val cleaner = createCleaner(ptr, ::mdb_env_close)

(Note: you will need to optIn for experimental stdlib API, so it’s up to you to decide on aproach)

[katokay]

I was thinking this was probably the case. I’ll try some of the suggestions here and report back how it goes. Thanks!

[katokay]

That did resolve the issue. Curious, is there a “correct” way to deal with when the native lib frees the handle? I would assume the ptr variable would then have nothing to do and the cleaner in that scenario wouldn’t be necessary.

[Landry Norris]

You allocated the ptr, so if you don't free it, you leaked 8 bytes.

[katokay]

To be specific, mdb_env_close when called, if free is subsequently called it will throw.

[Landry Norris]

The cleaner allows the memory to be freed when the memory is GC'ed. If you free it manually before then you don't need to worry about it.

[Landry Norris]

Will the free throw? It's typically undefined behavior unless the library detects double free itself.

[katokay]

throw was a wrong choice of words, the process crashes

[katokay]

Here is from the lib’s docs.
“Attempts to use any such handles after calling this function will cause a SIGSEGV. The environment handle will be freed and must not be used again after this call.”

[Landry Norris]

It's a C library, so you just have to make sure free happens exactly once regardless of code path. Cleaner is a good way to ensure this, but if you have another mechanism, it can be used instead.

[katokay]

Got it, makes sense.

[Landry Norris]

Does autoclosable ensure close only happens once? If not, I'd add a boolean isClosed and skip close if it's true.

[katokay]

Yep, that was exactly what I did.

[katokay]

It looks like that was the approach of another jvm library using the same native lib as well. Just wasn’t sure if there were unique options available on the KN cinterop way of things.

[katokay]

Because the function MUST be called, so it seems like freeing the handle yourself is removed as an option.

[Landry Norris]

For K/N, we have cleaner (JVM has Cleaner only on newer Android versions).

[katokay]

Thanks again for all the info.