Hey! Last weeks I’ve been working on researching m...

# feed

[Oleg Yukhnevich]

Hey!
Last weeks I’ve been working on researching multiplatform possibilities to provide access to native libraries (https://youtrack.jetbrains.com/issue/KT-39144), mainly for future use in https://github.com/whyoleg/cryptography-kotlin (which BTW will have first release soon)
https://twitter.com/why_oleg/status/1631223213110829058
And here are some results:
• kotlinx.cinterop inspired common API
• Works on Native, JVM(JNI/Panama), Android(JNI), WASM, JS
• manual bindings to part of libcrypto API
Next: automated bindings generation, API completness
Code can be found here: https://github.com/whyoleg/ffi-kotlin
Does someone is interested to contribute? :)
P.S. in thread there are screenshots for comparison of user-facing APIs for `ffi-kotlin`(fully common code) and `kotlinx.cinterop`(native only)

[Oleg Yukhnevich]

kotlinx.cinterop

[Oleg Yukhnevich]

ffi-kotlin

[svyatoslav.scherbina]

Hi!

kotlinx.cinterop inspired common API

While kotlinx.cinterop is a fine baseline in a sense (because it is known to be able to represent many of the required concepts, and the unsupported concepts are more or less known), it might make sense to start from scratch, just to avoid being affected by the current (arguable and opinionated) design and to properly take platform specifics into account.

Panama

Panama includes https://openjdk.org/jeps/424, which somewhat restricts operations with unsafe memory segments (see “Unsafe memory segments” and “Safety” sections). In particular, memory access operations on arbitrary CPointer can only be implemented using restricted FFM API.

Do you use FFM API? How do you overcome those restrictions?

• requires compiler to build JNI library - how to set up it properly to support different OSs?

• is it possible to use K/N provided clang with cross-compile possibilities here?

(from https://github.com/whyoleg/ffi-kotlin#jvmandroidjni)
You can start with run_konan clang clang $target from the Kotlin/Native distribution, e.g. this

~/.konan/kotlin-native-prebuilt-macos-x86_64-1.8.10/bin/run_konan clang clang linux_x64 -c 1.c

compiles 1.c to 1.o (and prints the whole clang command as well). Doesn’t play nicely with a linker, but you can fix this by adding -fuse-ld=lld compiler flag:

~/.konan/kotlin-native-prebuilt-macos-x86_64-1.8.10/bin/run_konan clang clang linux_x64 1.c -fuse-ld=lld

[Oleg Yukhnevich]

@svyatoslav.scherbina
1. can you explain more about what’s wrong with cinterop API, and what would you change if something? Overall only working with strings and structs is somehow different per platforms, while everything else is super similar and fit well into cinterop api
2. Panama implementation uses https://openjdk.org/jeps/434 (jep for jdk 20 with some changes to API) and yes I use FFM API, but Im not sure of what restrictions specifically you are talking, can you give an example?
3. Yeah, I tried using run_conan, but had issues with linker - so I will try with what you propose, thx!
4. If using konan, is it possible to download it from gradle without adding native targets, f.e. if there is no konan on user PC, and ffi used only for jvm?

[svyatoslav.scherbina]

can you explain more about what’s wrong with cinterop API, and what would you change if something? Overall only working with strings and structs is somehow different per platforms, while everything else is super similar and fit well into cinterop api

Overall, cinterop was designed
1. with language limitations in mind, for example, lack of value types, implicit conversions etc.
2. driven by really peculiar examples, like POSIX.
3. with attempts on making simple examples shorter, without real scalability here; this also made it inconsistent to some extent.
4. targeting people who know well both C and Kotlin
5. with attempts to be able to represent C concepts as correctly as possible (to the degree where it became unnatural to both C and Kotlin)

cinterop introduces really convoluted concepts (often to workaround 1, like CValuesRef).

As a result, cinterop is pretty hard to use. It is not intuitive: with a specific goal in mind (like use a C library in Kotlin, having C examples for it), one might have a hard time figuring out how to achieve it.

It cinterop doesn’t provide a way to write some of the bindings manually, which could have made things simpler (by allowing auto-generator to be imprecise or non-universal in some cases).

I believe some of these things can be avoided even in a third-party project.

I use FFM API, but Im not sure of what restrictions specifically you are talking, can you give an example?

Quote from the JEP:

The FFM API can also produce unsafe segments, that is, memory segments whose spatial and temporal bounds are user-provided and cannot be verified by the Java runtime (see MemorySegment::ofAddress).

The unsafe methods in the FFM API do not pose the same risks as JNI functions; they cannot, e.g., change the values of final fields in Java objects. On the other hand, the unsafe methods in the FFM API are easy to call from Java code. For this reason, the use of unsafe methods in the FFM API is restricted: Their use is permitted but, by default, every such use causes a warning to be issued at run time.

CPointer is basically an unsafe segment (e.g. when it is returned from a C function).

if using konan, is it possible to download it from gradle without adding native targets, f.e. if there is no konan on user PC, and ffi used only for jvm?

I’m not aware of any stable easy approach for this.

[Oleg Yukhnevich]

1. Can we chat/make c call sometime to discuss ideas on how it will be possible to improve ffi? Dart BTW has similar API
2. For now my idea was to be able to easy migrate native cinterop -> multiplatform. though even now it differs in some places, but improving overall API is always good
3. Manual bindings for multiplatform will be even harder, as there are much more things involved
4. When CPointer returned from function, it can be exposed via Unbounded(Unsafe) segment, and you can do with it what you want. I haven’t seen any warnings during testing. Jextract uses the same approach for such declarations, so Im not sure, that there is some problem here, or at least I don’t understand it yet :)

[svyatoslav.scherbina]

Can we chat/make c call sometime to discuss ideas on how it will be possible to improve ffi? Dart BTW has similar API

Yes, sure. I’ve sent you a DM.

Manual bindings for multiplatform will be even harder, as there are much more things involved

A compiler plugin might help.

When CPointer returned from function, it can be exposed via Unbounded(Unsafe) segment,

Exactly. And the JEP says that unsafe segments are restricted and trigger a warning.