<@U02K3A6E6KD> - I have some questions about the k...

# kotlin-native

[napperley]

@Pavel Kunyavskiy [JB] - I have some questions about the knm (Kotlin Native Module) file format:

1. What is the current status on the knm file format?
2. Will the knm file format continue to exist?
3. How will the knm files work in their final form?
4. What is being done to remove the messy comments that make the file format very verbose/noisy (impossible to parse)?
5. Is the knm file format being positioned for use in code generation (eg to create Kotlin bindings for a C library)?
6. What improvements are being made to the IntelliJ/Fleet Kotlin plugin(s) to support the knm file format (@Bart)?
7. Will there be an official Kotlin library (@e5l) provided to aid in extracting meta data from knm files (eg for use in creating Kotlin bindings for a C library)?

[napperley]

Below is an example of what a knm file looks like in its current form (trimmed for brevity):

// IntelliJ API Decompiler stub source generated from a class file
// Implementation of methods is not available

package io.gitlab.embedSoft.lvglKt.core.clib.lvgl

@kotlinx.cinterop.internal.CStruct public final class lv_mem_monitor_t public constructor(rawPtr: kotlinx.cinterop.NativePtr /* = kotlin.native.internal.NativePtr */) : kotlinx.cinterop.CStructVar {
    @kotlinx.cinterop.internal.CStruct.VarType @kotlin.Deprecated public companion object : kotlinx.cinterop.CStructVar.Type {
    }

    public final var frag_pct: platform.posix.uint8_t /* = kotlin.UByte */ /* compiled code */

    public final var free_biggest_size: platform.posix.uint32_t /* = kotlin.UInt */ /* compiled code */

    public final var free_cnt: platform.posix.uint32_t /* = kotlin.UInt */ /* compiled code */

    public final var free_size: platform.posix.uint32_t /* = kotlin.UInt */ /* compiled code */

    public final var max_used: platform.posix.uint32_t /* = kotlin.UInt */ /* compiled code */

    public final var total_size: platform.posix.uint32_t /* = kotlin.UInt */ /* compiled code */

    public final var used_cnt: platform.posix.uint32_t /* = kotlin.UInt */ /* compiled code */

    public final var used_pct: platform.posix.uint8_t /* = kotlin.UByte */ /* compiled code */
}

public final class lv_mem_buf_t public constructor(rawPtr: kotlinx.cinterop.NativePtr /* = kotlin.native.internal.NativePtr */) : kotlinx.cinterop.CStructVar {
    @kotlinx.cinterop.internal.CStruct.VarType @kotlin.Deprecated public companion object : kotlinx.cinterop.CStructVar.Type {
    }

    public final var p: kotlinx.cinterop.COpaquePointer? /* = kotlinx.cinterop.CPointer<out kotlinx.cinterop.CPointed>? */ /* compiled code */

    public final var size: platform.posix.uint16_t /* = kotlin.UShort */ /* compiled code */

    public final var used: platform.posix.uint8_t /* = kotlin.UByte */ /* compiled code */
}

[Pavel Kunyavskiy [JB]]

Sorry, I'm not available until next Monday. This chat is checked by different people from team. Tagging me explicitly significantly reduce chance, that someone else will help you, if I'm not available.

[napperley]

@Pavel Kunyavskiy [JB] - Who should I be contacting?

[Pavel Kunyavskiy [JB]]

Sorry, we don't have dedicated Native support engineer now.

[napperley]

That is unfortunate. Would that issue be resolved sometime next year?

[napperley]

Is there some hidden documentation on the knm file format?

[e5l]

Cc @Vsevolod Tolstopyatov [JB]

[svyatoslav.scherbina]

Who should I be contacting?

Support engineer is not a person who can answer all these questions.
Please don’t tag anyone. As Pavel said, that reduces the chance that someone else will help you.

Is there some hidden documentation on the knm file format?

No hidden documentation, please use the source code, it is open.

What is the current status on the knm file format?

It is supported by the compiler and various tools.

Will the knm file format continue to exist?

No guarantees, likely yes.

How will the knm files work in their final form?

The files don’t “work”, they can be emitted or parsed. Your question is unclear.

What is being done to remove the messy comments that make the file format very verbose/noisy (impossible to parse)?

There are no messy comments in the knm file format.

Below is an example of what a knm file looks like in its current form (trimmed for brevity):

That’s not true. knm is a binary format, what you show is an output of a tool that can parse knm format and dump it in a human-readable form. Please create a YouTrack issue if you have problems with it or would like to request a feature.

Is the knm file format being positioned for use in code generation (eg to create Kotlin bindings for a C library)?

The knm file format is used as an implementation detail of klibs, and cinterop tool in particular. Not sure what you mean by positioning, but klibs produced by cinterop can be read by the compiler, including its current and likely next versions.

Will there be an official Kotlin library (@e5l) provided to aid in extracting meta data from knm files (eg for use in creating Kotlin bindings for a C library)?

You can try to use this library: https://repo.maven.apache.org/maven2/org/jetbrains/kotlinx/kotlinx-metadata-klib/
It is experimental, no guarantees.

(eg for use in creating Kotlin bindings for a C library)?

Why would you need this, if you have cinterop?

[napperley]

Why would you need this, if you have cinterop?

When getting a wrapper generated for a C library the cinterop tool works fine, but isn't sufficient for bindings, which need to provide proper Kotlin APIs instead of C like APIs.

You can try to use this library: https://repo.maven.apache.org/maven2/org/jetbrains/kotlinx/kotlinx-metadata-klib/

Is there an example of using the KotlinX Metadata Klib library (aka, sample project)?

No hidden documentation, please use the source code, it is open.

Is there a good starting point in the source code for reference documentation that introduces the knm file format?

[svyatoslav.scherbina]

When getting a wrapper generated for a C library the cinterop tool works fine, but isn’t sufficient for bindings, which need to provide proper Kotlin APIs instead of C like APIs.

Why not write these Kotlin APIs in Kotlin then? You can’t actually generate Kotlin wrappers as .knm, because .knm doesn’t contain function bodies.

Is there an example of using the KotlinX Metadata Klib library (aka, sample project)?

No.

Is there a good starting point in the source code for reference documentation that introduces the knm file format?

https://github.com/JetBrains/kotlin

git grep -r knm

[napperley]

At the end of the day the cinterop tool is still needed to create wrappers for the C APIs, which is the first step for creating Kotlin bindings. The next steps involve generating, or writing by hand Kotlin APIs that use the C APIs in wrapper form. Without the wrappers the C APIs can't be used in Kotlin code. Bindings for a library usually mean providing a carbon copy (or one that is very similar, which is more likely) of the APIs in a form that is suitable (idiomatic) for the target language; will be Kotlin in this case. Often code generation is used to create the bindings, hence access to meta data for the wrapper is required.

Function bodies aren't important for bindings. It is the function signatures that are important. Also additional metadata would need to be provided by the C library for other important things like memory lifetimes, fake namespaces used etc.

[napperley]

I am about to show a hypothetical example of what I mean about Kotlin bindings using the LVGL library ( https://lvgl.io/ ). Below is the C code snippet:

// ...

static void event_handler(lv_event_t * e)
{
    lv_event_code_t code = lv_event_get_code(e);

    if(code == LV_EVENT_CLICKED) {
        LV_LOG_USER("Clicked");
    }
    else if(code == LV_EVENT_VALUE_CHANGED) {
        LV_LOG_USER("Toggled");
    }
}

void lv_example_btn_1(void)
{
    lv_obj_t * label;

    lv_obj_t * btn1 = lv_btn_create(lv_scr_act());
    lv_obj_add_event_cb(btn1, event_handler, LV_EVENT_ALL, NULL);
    lv_obj_align(btn1, LV_ALIGN_CENTER, 0, -40);

    label = lv_label_create(btn1);
    lv_label_set_text(label, "Button");
    lv_obj_center(label);

    lv_obj_t * btn2 = lv_btn_create(lv_scr_act());
    lv_obj_add_event_cb(btn2, event_handler, LV_EVENT_ALL, NULL);
    lv_obj_align(btn2, LV_ALIGN_CENTER, 0, 40);
    lv_obj_add_flag(btn2, LV_OBJ_FLAG_CHECKABLE);
    lv_obj_set_height(btn2, LV_SIZE_CONTENT);

    label = lv_label_create(btn2);
    lv_label_set_text(label, "Toggle");
    lv_obj_center(label);
}

Below is the Kotlin snippet using the hypothetical Kotlin bindings for the LVGL library:

// ... 

fun eventHandler(evt: Event) {
	val code = evt.code
	if (code == EventType.CLICKED) logUser("Clicked")
	else if (code == EventType.VALUE_CHANGED) logUser("Toggled")
}

fun lvExampleBtn1() {
	val btn1 = button(actualScreen()) {
		addEventCallback(::eventHandler, EventType.ALL)
		alignObject(alignType = AlignmentType.CENTER, xPos = 0, 
			yPos = -40)
	}
	label(btn1) {
		text = "Button"
		centerObject()
	}
	val btn2 = button(actualScreen()) {
		addEventCallback(::eventHandler, EventType.ALL)
		alignObject(alignType = AlignmentType.CENTER, xPos = 0, 
			yPos = 40)
		addFlag(ObjectFlag.CHECKABLE)
		height = SIZE_CONTENT
	}
	label(btn2) {
		text = "Toggle"
		centerObject()
	}
}

[svyatoslav.scherbina]

Alright, and how can the compiler now that actualScreen should call lv_scr_act? That’s what function bodies are required for.

[napperley]

Why are the function bodies required? In the example above lv_scr_act (in the LVGL documentation the lv_scr_act function is used to get the active screen of the default display - https://docs.lvgl.io/latest/en/html/overview/display.html?highlight=lv_scr_act#_CPPv410lv_scr_actv ) can't be used since it doesn't fit into the Kotlin Coding Conventions with the naming (also uses ambiguous abbreviations), hence the name actualScreen is used (should have been renamed to activeScreen) for the function name instead. Since cases like this will crop up from time to time, there will need to be a facility in the code generation system (for creating the bindings) to do manual overrides. There is no such thing as perfect code generation. Many code generation runs, and examination/tweaking of the results will need to be done , before ending up a good result that is idiomatic Kotlin and provides a decent develop experience for the user with the API ergonomics.

One cannot expect the code generation to pick up everything that is needed to generate usable bindings for a library. There will be a few places where manual overrides are needed. Often key stuff like memory management with APIs are picked up through library documentation, unless the library has a meta data system that can be used to collect this data by symbol lookup.

[svyatoslav.scherbina]

Well, maybe for lv_scr_act you can simply change the function name, but this won’t work for cases like addEventCallback, where you need to have a wrapper with non-trivial logic inside. Otherwise the compiler just doesn’t know how to translate addEventCallback(::eventHandler, EventType.ALL) into lv_obj_add_event_cb(this, staticCFunction(::eventHandler), EventType.ALL, null)
Alternatively, you can do this without generating Kotlin code with the help of a compiler plugin.

[napperley]

Are you suggesting that there are certain parts of the code generation that would be better covered with a compiler plugin? If that is the case then which use cases/areas would a compiler plugin cover when it comes to generating Kotlin bindings for a C library?

[napperley]

One question I should have mentioned before the others is, Are Kotlin compiler plugins suitable for cases where Kotlin bindings need to be generated for a C library?

[svyatoslav.scherbina]

Are you suggesting that there are certain parts of the code generation that would be better covered with a compiler plugin?

No, I’m just saying that you can usually make a compiler plugin that generates wrappers instead of writing these wrappers manually.

Are Kotlin compiler plugins suitable for cases where Kotlin bindings need to be generated for a C library?

For some parts of the process, maybe.