Hey, is there someone who can help with cinterop a...

# kotlin-native

[Oleg Yukhnevich]

Hey, is there someone who can help with cinterop and pointers ? 🙂
there is external declaration:
fun EVP_MAC_CTX_set_params(ctx: CValuesRef<EVP_MAC_CTX>?, params: CValuesRef<OSSL_PARAM>?): <http://kotlin.Int|kotlin.Int> (params is a reference to an array)
example of usage in C is here https://www.openssl.org/docs/man3.0/man3/EVP_MAC_init.html (in the end of the page).
I’ve tried to something like this, and it compiles and works without errors on runtime, but looks like it doesn’t work as expected, as those params are not visible by `openssl`…

val mac = EVP_MAC_fetch(null, "HMAC", null)
val context = EVP_MAC_CTX_new(mac)

val params = allocArrayOf(
    OSSL_PARAM_construct_utf8_string("digest", "SHA256".cstr, 0).ptr,
    OSSL_PARAM_construct_end().ptr
)
check(EVP_MAC_CTX_set_params(context, params[0]) == 1) // works, returns 1

check(EVP_MAC_CTX_get_mac_size(context) > 0) // fails, but should not

The error is error:0300009F:digital envelope routines::message digest is null - so looks like it don’t see digest key in params array for some reason
Where I could dig to understand what I do wrong? Thx in advance! 🙂

[ephemient]

the cstr doesn't live long enough. OSSL_PARAM_construct_utf8_string doesn't copy it, it holds onto the pointer

[Oleg Yukhnevich]

any idea on how to overcome this? I’ve tried pinning, but looks like it doesn’t help here

[Oleg Yukhnevich]

I’ve added such declaration to def file (the same should work for EVP_MAC_CTX_set_params):

static inline int hmac_init(EVP_MAC_CTX *ctx, const unsigned char *key, size_t keylen, const char *digest_name) {
    OSSL_PARAM params[4], *p = params;
    *p++ = OSSL_PARAM_construct_utf8_string("digest", digest_name, 0);
    *p = OSSL_PARAM_construct_end();
    return EVP_MAC_init(ctx, key, keylen, params);
}

and using like:

hmac_init(context, key.asUByteArray().refTo(0), key.size.convert(), hashAlgorithm)

it works as expected
While I can leave with it, I literally don’t understand, why it doesn’t work with kotlin constructions…

[napperley]

Every C library has a different way of dealing with "fake" strings. Have you run into design issues with the OpenSSL API where its string handling is all over the place?

[napperley]

How does the OpenSSL library handle memory management (incl with strings)?

[Oleg Yukhnevich]

TBH, I don’t know 🙂
As Im not really a good C developer (or even not at all a C developer)

[Oleg Yukhnevich]

And I found one more thing, that is hard to understand
the issue is that while this code works as expected:

val params = allocArrayOf(
  OSSL_PARAM_construct_utf8_string("digest".cstr.ptr, hashAlgorithm.cstr.ptr, 0).ptr,
  OSSL_PARAM_construct_end().ptr
)
EVP_MAC_init(context, key.asUByteArray().refTo(0), key.size.convert(), params[0])

But in some conditions (looks like when GC works, or anything else) this code starts to fail with seg fault
So I tried once again using C code in def file:

static inline int EVP_MAC_init_HMAC(EVP_MAC_CTX *ctx, const unsigned char *key, size_t keylen, const char *digest_name) {
     OSSL_PARAM params[2], *p = params;
     *p++ = OSSL_PARAM_construct_utf8_string("digest", digest_name, 0);
     *p = OSSL_PARAM_construct_end();
     return EVP_MAC_init(ctx, key, keylen, params);
 }

and on call site:

EVP_MAC_init_HMAC(context, key.asUByteArray().refTo(0), key.size.convert(), hashAlgorithm)

This way it works all time. (and look, here we pass hashAlgorithm: String as plain string and it works as expected
for reference commit with this changes: https://github.com/whyoleg/kcwrapper/commit/93dec2fbbb4b74499684b1addce5b92519655c0d
So I would think that is some issue in K/N interop mappings generation or I’m missing some obvious thing.
@svyatoslav.scherbina may be you can somehow help with understanding: is it an issue on my side, or inside K/N interop and I should fill an issue?

[svyatoslav.scherbina]

return EVP_MAC_init(ctx, key, keylen, params);

So, in C version you pass the pointer to the params array, while in Kotlin

EVP_MAC_init(context, key.asUByteArray().refTo(0), key.size.convert(), params[0])

params[0] is not a pointer to the array, but the first value instead, i.e. OSSL_PARAM_construct_utf8_string("digest".cstr.ptr, hashAlgorithm.cstr.ptr, 0).ptr. Please try replacing params[0] with params to make it equivalent to C.

[Oleg Yukhnevich]

I’ve tried passing params.reinterpret() and just casting (because types are different), but in this case it doesn’t work at all (init returns 0, means failure)
I will point one time, that when passing params[0] code works, but under stress test (which create a lot of keys and then try to sign with them) it starts to FAIL with seg fault at random moment specifically in call to EVP_MAC_init. while using C code in def - everything works

For context
• OSSL_PARAM_construct_utf8_string returns CValue<OSSL_PARAM> (https://www.openssl.org/docs/man3.0/man3/OSSL_PARAM_construct_utf8_string.html)
• params in kotlin is of type CPointer<CPointerVarOf<CPointer<OSSL_PARAM>>>
• EVP_MAC_init accepts for params CValuesRef<OSSL_PARAM> (in openssl header it’s OSSL_PARAM[], https://www.openssl.org/docs/man3.0/man3/EVP_MAC_init.html)

[svyatoslav.scherbina]

Your C code creates an array of two struct values, and passes the pointer to that array (i.e. to the first element) to EVP_MAC_init.
Your Kotlin code creates an array of two pointers to struct values, and passes the first of them to EVP_MAC_init. So, as a result, EVP_MAC_init gets a pointer to OSSL_PARAM_construct_utf8_string("digest", hashAlgorithm.cstr.ptr, 0), followed by arbitrary garbage instead of OSSL_PARAM_construct_end(). Sometimes it gets lucky, and that garbage works as a terminator, sometimes not.

So, you need to start with val params = allocArray<OSSL_PARAM>(2), initialize the elements properly and pass params to EVP_MAC_init. Or, alternatively, use val params = createValues<OSSL_PARAM>(2) { ... }.

[Oleg Yukhnevich]

Thx for help!
I’ve tried already to do something like this, but the problem is, that I wanted to use OSSL_PARAM_construct_utf8_string declaration from openssl, and not construct structs by my self (error prone)
And so I haven’t found any information/documentation about how to allocate array of structs, and then put in it already created struct values…
But!
Looks like I found a way (after several tries on finding what I need):

val params = allocArray<OSSL_PARAM>(2)
OSSL_PARAM_construct_utf8_string("digest".cstr.ptr, hashAlgorithm.cstr.ptr, 0).place(params[0].ptr)
OSSL_PARAM_construct_end().place(params[1].ptr)
EVP_MAC_init(context, key.asUByteArray().refTo(0), key.size.convert(), params)

And this works as expected!
Big thanks for help!

P.S. I think, that API or at least documentation around creating arrays of structs could be improved 🙂

[svyatoslav.scherbina]

Yes, that should work.

P.S. I think, that API or at least documentation around creating arrays of structs could be improved 🙂

That is totally correct 🙂

[Oleg Yukhnevich]

Should I fill an issue on youtrack for documentation or even minimal POC API for this?

[svyatoslav.scherbina]

Yes, that would be helpful, thank you!

[Oleg Yukhnevich]

BTW, while I understand, that may be it’s not strictly your area @svyatoslav.scherbina but, is there any plans to release cinterop for JVM/WASM ?
I know, that JVM flavor exists to power cinterop clang bindings, but no doc for it of course :)

Some time in a future I will want to port those openssl bindings to JVM (via jni or panama)
And so I will be interested even on working to integrate cinterop with JVM similar to how we have it for K/N (I just like hard interesting tasks, hah)

[svyatoslav.scherbina]

is there any plans to release cinterop for JVM/WASM ?

We have no particular plans for this.
As you probably know, we have a YouTrack issue for this feature request: https://youtrack.jetbrains.com/issue/KT-39144/Generate-C-bindings-for-all-platforms-through-cinterop (just adding it here for visibility).

I know, that JVM flavor exists to power cinterop clang bindings, but no doc for it of course :)

Missing docs is not the main problem here. There are a lot of complicated integration tasks to be solved before this can become useful. Also, even the existing basic implementation for JVM is somewhat behind. And that implementation we have should probably be ported to panama.

[Oleg Yukhnevich]

We have no particular plans for this.

okay..

yes, I know about an issue, still thought that ‘speculation’ talks are better to perform in slack to not pollute an issue discussion 🙂

And that implementation we have should probably be ported to panama.

just wanted to mention, in ideal case I would think that there should be a choice: use JNI or use Panama, as there is no Panama support in Android. So to be able to build 2 different JARs (or klibs?) or multi-release JAR will be useful

There are a lot of complicated integration tasks to be solved before this can become useful. Also, even the existing basic implementation for JVM is somewhat behind

Yeah, I understand, that it’s not easy… still, if implementation of something like this will be started some day, i’m willing to help!

[Oleg Yukhnevich]

created an issue regarding structs and arrays: https://youtrack.jetbrains.com/issue/KT-56570/CInterop-Improve-experience-when-working-with-arrays-of-structs

[svyatoslav.scherbina]

Thank you!

[Oleg Yukhnevich]

Hey @svyatoslav.scherbina, regarding cinterop for JVM (and not only)
I’ve done some research on multiplatform cinterop - results are here https://kotlinlang.slack.com/archives/C0BJ0GTE2/p1677748948814649