How do I build a project with cinterops for multip...

# multiplatform

[martmists]

How do I build a project with cinterops for multiple platforms? adding cinterops to all targets results in it failing to build cinterops for windows on linux (build.gradle.kts and error log in thread)

[martmists]

My build.gradle.kts:

plugins {
    kotlin("multiplatform")
    `maven-publish`
}

kotlin {
    val targets = listOf(
        linuxX64(),
        linuxArm64(),
        mingwX64(),
        mingwX86(),
        macosX64(),
        macosArm64(),
    )

    sourceSets {
        val nativeMain by creating {

        }

        val linuxX64Main by getting {
            dependsOn(nativeMain)
        }

        val linuxArm64Main by getting {
            dependsOn(nativeMain)
        }

        val mingwX64Main by getting {
            dependsOn(nativeMain)
        }

        val mingwX86Main by getting {
            dependsOn(nativeMain)
        }

        val macosX64Main by getting {
            dependsOn(nativeMain)
        }

        val macosArm64Main by getting {
            dependsOn(nativeMain)
        }
    }

    targets.forEach {
        it.apply {
            val main by compilations.getting {

            }
            val python by main.cinterops.creating {
                // this cinterop is breaking builds
            }

            binaries {
                staticLib {
                    binaryOptions["memoryModel"] = "experimental"
                    freeCompilerArgs += listOf("-Xgc=cms")
                }
            }
        }
    }
}
buildConfig {
    packageName.set("com.martmists.kpy.cfg")

    buildConfigField("String", "VERSION", "\"${project.version}\"")
}

val generatePythonDef = tasks.create<Exec>("generatePythonDef") {
    val minPyVersion = "3.9.0"

    group = "interop"
    description = "Generate Python.def file"
    executable = "python3"

    val cinteropDir = "${project.projectDir.absolutePath}/src/nativeInterop/cinterop"
    val parts = minPyVersion.split(".").toMutableList()
    while (parts.size < 4) {
        parts.add("0")
    }
    val versionHex = parts.joinToString("") { it.toInt().toString(16) }

    args(
        "-c",
        """
import sysconfig
paths = sysconfig.get_paths()
template = '''
headers = Python.h
package = python
compilerOpts = -I"{INCLUDE_DIR}"
linkerOpts = -L"{LIB_DIR}" -l python3

---

struct KtPyObject {{
    PyObject base;
    void* ktObject;
}};

// Wrapper func for _PyUnicode_AsString macro
char* PyUnicode_AsString(PyObject* obj) {{
    return _PyUnicode_AsString(obj);
}}
'''.strip()
with open('${cinteropDir.replace('\\', '/')}/python.def', 'w') as fp:
    fp.write(template.format(
        INCLUDE_DIR=paths['platinclude'],
        LIB_DIR='/'.join(paths['platstdlib'].split('/')[:-1]),
        MIN_VERSION_HEX='0x${versionHex}'
    ))
        """.trim()
    )
    outputs.upToDateWhen { false }
}

for (target in listOf("LinuxX64", "LinuxArm64", "MacosX64", "MacosArm64", "MingwX64", "MingwX86")) {
    tasks.getByName("cinteropPython${target}") {
        dependsOn(generatePythonDef)
    }
}

if (project.ext.has("mavenToken")) {
    publishing {
        repositories {
            maven {
                name = "Host"
                url = uri("<https://maven.martmists.com/releases>")
                credentials {
                    username = "admin"
                    password = project.ext["mavenToken"]!! as String
                }
            }
        }

        publications.withType<MavenPublication> {

        }
    }
} else if (System.getenv("CI") == "true") {
    publishing {
        repositories {
            maven {
                name = "Host"
                url = uri(System.getenv("GITHUB_TARGET_REPO")!!)
                credentials {
                    username = "kpy-actions"
                    password = System.getenv("DEPLOY_KEY")!!
                }
            }
        }

        publications.withType<MavenPublication> {
            if (System.getenv("DEPLOY_TYPE") == "snapshot") {
                version = System.getenv("GITHUB_SHA")!!
            }
        }
    }
}

[martmists]

Error log:

5:31:31 PM: Executing 'build'...


> Configure project :kpy-library
Kotlin Multiplatform Projects are an Alpha feature. See: <https://kotlinlang.org/docs/reference/evolution/components-stability.html>. To hide this message, add 'kotlin.mpp.stability.nowarn=true' to the Gradle properties.


Some Kotlin/Native targets cannot be built on this linux_x64 machine and are disabled:
    * In project ':kpy-library':
        * targets 'macosX64', 'macosArm64' (can be built with one of the hosts: macos_x64, macos_arm64)
To hide this message, add 'kotlin.native.ignoreDisabledTargets=true' to the Gradle properties.


> Task :kpy-library:generateProjectStructureMetadata
> Task :commonizeNativeDistribution
> Task :kpy-library:generatePythonDef
> Task :kpy-library:cinteropPythonLinuxArm64 UP-TO-DATE
> Task :kpy-library:cinteropPythonLinuxX64 UP-TO-DATE
> Task :kpy-library:cinteropPythonMacosArm64 SKIPPED
> Task :kpy-library:cinteropPythonMacosX64 SKIPPED

> Task :kpy-library:cinteropPythonMingwX64 FAILED

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See <https://docs.gradle.org/7.4.2/userguide/command_line_interface.html#sec:command_line_warnings>
6 actionable tasks: 4 executed, 2 up-to-date
Exception in thread "main" java.lang.Error: /usr/include/python3.10/pyport.h:235:10: fatal error: 'sys/select.h' file not found
	at org.jetbrains.kotlin.native.interop.indexer.UtilsKt.ensureNoCompileErrors(Utils.kt:273)
	at org.jetbrains.kotlin.native.interop.indexer.IndexerKt.indexDeclarations(Indexer.kt:1189)
	at org.jetbrains.kotlin.native.interop.indexer.IndexerKt.buildNativeIndexImpl(Indexer.kt:1178)
	at org.jetbrains.kotlin.native.interop.indexer.IndexerKt.buildNativeIndexImpl(Indexer.kt:1174)
	at org.jetbrains.kotlin.native.interop.gen.jvm.DefaultPlugin.buildNativeIndex(Plugins.kt:33)
	at org.jetbrains.kotlin.native.interop.gen.jvm.MainKt.processCLib(main.kt:272)
	at org.jetbrains.kotlin.native.interop.gen.jvm.MainKt.interop(main.kt:76)
	at org.jetbrains.kotlin.cli.utilities.InteropCompilerKt.invokeInterop(InteropCompiler.kt:45)
	at org.jetbrains.kotlin.cli.utilities.MainKt.mainImpl(main.kt:38)
	at org.jetbrains.kotlin.cli.utilities.MainKt.main(main.kt:60)

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':kpy-library:cinteropPythonMingwX64'.
> Process 'command '/usr/lib/jvm/java-17-openjdk/bin/java'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at <https://help.gradle.org>

BUILD FAILED in 2s
5:31:34 PM: Execution finished 'build'.

[Anton Lakotka [JB]]

Exception in thread "main" java.lang.Error: /usr/include/python3.10/pyport.h:235:10: fatal error: 'sys/select.h' file not found

this means that you need to provide headers of MSYS2 https://packages.msys2.org/package/msys2-runtime-devel
But it also might require more packages, basically, you should be able to build python from sources on your host machine for Windows.
So you need to read how to do that, what packages are required and so on.
You can also try reducing the scope of symbols that are needed for your library. You have imported Python.h which imports lots of symbols, perhaps you need less of them 🤷

[martmists]

Is there a way to limit cinterops to only build on the "matching" host only?

[Anton Lakotka [JB]]

You can do that, Jut add some if before

val python by main.cinterops.creating

but keep in mind that your compilation would still fail. because of missing declarations.

So if you want to build your lib then you need to provide correct headers.
I downloaded zst package of python3 headers from msys2: https://packages.msys2.org/package/mingw-w64-x86_64-python3.10?repo=mingw64

I unpacked it into project root directory. See screenshot.

Then I created separate def file for windows-target.

headers = Python.h
package = python
compilerOpts = -I"msys2-python/mingw64/include/python3.10"
linkerOpts = -L"/usr/lib" -l python3

---

struct KtPyObject {
    PyObject base;
    void* ktObject;
};

// Wrapper func for _PyUnicode_AsString macro
char* PyUnicode_AsString(PyObject* obj) {
    return _PyUnicode_AsString(obj);
}

Then I updated build.gradle.kts to use that def file for windows:

val python by main.cinterops.creating {
    if (nativeTarget.konanTarget != KonanTarget.MINGW_X64) {
        defFile = project.file("src/nativeInterop/cinterop/python.def")
    } else {
        defFile = project.file("src/nativeInterop/cinterop/pythonMingwX64.def")
    }
}

And I managed to generate cinterops for windows without any issue.

But MinGW target have a different number lengths and common code cannot be compiled for all three platforms.
Thus you need to use convert function that solves this issue: https://github.com/JetBrains/kotlin/blob/6a670dc5f38fc73eb01d754d8f7c158ae0176ceb/[…]terop/Runtime/src/native/kotlin/kotlinx/cinterop/NativeUtils.kt

And new code would look like this (see 2nd screenshot)

Having all that you can publish your library as it should. Of course, keep in mind that for publishing MacOS target you would need macos host.

[martmists]

Then I created separate def file for windows-target.

I don't think that's possible here as the linux, mac and windows locations of the python libraries are in different locations, and you can't link the windows one against the python one, hence why I have a task to generate the def files.

but keep in mind that your compilation would still fail. because of missing declarations.

Would commonizing the cinterop not handle that properly?

[martmists]

Attempting to install the msys2 package in github actions also fails:

Error: MSYS2 does not work on non-windows platforms; please check the 'runs-on' field of the job

And I can't think of a reasonable way to extract the zst package as gradle task as it'd require a lot of html scraping to get the download link and/or hardcoding the download URLs per python version we support.

[Anton Lakotka [JB]]

I don't think that's possible here as the linux, mac and windows locations of the python libraries are in different locations, and you can't link the windows one against the python one, hence why I have a task to generate the def files.

This one is clear.
But you need to keep in mind that when you link/compile on Linux host to Windows binary you need to provide Windows-specific headers & binaries of your dependencies (i.e Python).
The trick with headers files you did would work only when you compile for Windows on Windows, for Linux on Linux and etc.
Kotlin/Native allows you to compile on foreigner targets (except Apple-specific). That is why I had to use different def file for windows target and point to windows-specific headers.

Would commonizing the cinterop not handle that properly?

If you declare cinterop only on the host-related target. And other targets wouldn't have cinterops dependencies. Then commonizer would not be able to find "common" API between such targets.
Since commonizer roughly does the intersection of APIs and intersection between empty set and something else is always an empty set. There are some experiments of making commonizer be more friendly and allowing cases when cinterop not available on some hosts.
But it was just an experiment without any usable outputs yet.

[Anton Lakotka [JB]]

Attempting to install the msys2 package in github actions also fails:

you don't have to install msys2 package, you just need to download .zst archive by direct link.
You also don't need to scrap anything just use the link:

<https://mirror.msys2.org/mingw/mingw64/mingw-w64-x86_64-python3.10-3.10.2-3-any.pkg.tar.zst>

You can also build it up like this:

fun pythonHeadersForMingw(version: String) = "<https://mirror.msys2.org/mingw/mingw64/mingw-w64-x86_64-python$version-any.pkg.tar.zst>"

And then extract it like this:

tar --use-compress-program=unzstd -xvf mingw-w64-x86_64-python3.10-3.10.2-3-any.pkg.tar.zst

Think about it as preparation steps in order to build some native libraries/applications. Like installing some *-dev packages.
Since Gradle tooling doesn't support well different repositories such as msys2.
-----

Another option would be to publish MinGW library as a separate project. Which would require some other tricks. And I can't currently figure out how to do that in a better way.
But if you are interested I can come up with some ideas.
But personally, I would go the path with using MSYS2 package since it worked well already on my machine.