Does Kotlin/Wasm provide APIs to deal with Wasm memory, import, exports? My use case would be to exp...

# webassembly

[sdeleuze]

Does Kotlin/Wasm provide APIs to deal with Wasm memory, import, exports? My use case would be to experiment on Kotlin/Wasm support in WIT bindgen, taking inspiration from the work already done on Java via the TeaVM WASI fork (demo: https://bytecodealliance.github.io/wit-bindgen/).

[Svyatoslav Kuzmich [JB]]

Not yet. It is in progress: https://github.com/JetBrains/kotlin/tree/rr/skuzmich/wasm-unsafe/libraries/stdlib/wasm/src/kotlin/wasm/unsafe

[Svyatoslav Kuzmich [JB]]

I’m experimenting with witx bindings right now. witx for WASI still has some raw pointers exposed that will have to be wrapped manually. Hopefully component model ABI will change it!

[sdeleuze]

Preview2 draft API is available in https://github.com/bytecodealliance/preview2-prototyping/blob/main/wit/wasi.wit

[sdeleuze]

The project allows to run it on top of a preview1 runtime

[sdeleuze]

Happy to contribute on wit-bindgen side once the APIs are available

[sdeleuze]

For Kotlin/Wasm experiments maybe worth using directly https://github.com/bytecodealliance/preview2-prototyping

[Svyatoslav Kuzmich [JB]]

Glad to hear that you are interested in contributing wit bindings! In case you are up for prototyping using source-build toolchain and APIs that might change (maybe a little bit) — please ping me, and I can prepare a branch with Memory and Import APIs

[Svyatoslav Kuzmich [JB]]

The branch with memory and import APIs is at https://github.com/JetBrains/kotlin/tree/skuzmich/wit-prototyping
Experiments with using this API to generate bindings for WASI preview1 https://github.com/skuzmich/kotlin-wasi-bindings

[sdeleuze]

@Svyatoslav Kuzmich [JB] Super excited by what you shared even if that's early stage. I guess I may try to use your unsafe Wasm API to work on adding Kotlin support to https://github.com/bytecodealliance/wit-bindgen and try to leverage https://github.com/bytecodealliance/preview2-prototyping to use WASI preview2 using your branch. May take some time to grow my skills on everything, but I guess I can try.

[sdeleuze]

After running the tests, TEST-kotlinx.wasi.WasiTests.xml contains:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="kotlinx.wasi.WasiTests" tests="5" skipped="0" failures="0" errors="0" timestamp="2022-12-25T19:44:57" hostname="m1" time="0.005">
  <properties/>
  <testcase name="Test_args_sizes_get[wasm, node]" classname="kotlinx.wasi.WasiTests" time="0.001"/>
  <testcase name="Test_args_get[wasm, node]" classname="kotlinx.wasi.WasiTests" time="0.001"/>
  <testcase name="Test_environ_sizes_get[wasm, node]" classname="kotlinx.wasi.WasiTests" time="0.0"/>
  <testcase name="Test_environ_get[wasm, node]" classname="kotlinx.wasi.WasiTests" time="0.0"/>
  <testcase name="testPrintln[wasm, node]" classname="kotlinx.wasi.WasiTests" time="0.002"/>
  <system-out><![CDATA[PATH=/usr/local/bin:/usr/bin
HOME=/Users/이준
WASI error: WasiError: WASI call failed with INVAL
WASI error: WasiError: WASI call failed with INVAL
WASI error: WasiError: WASI call failed with INVAL
WASI error: WasiError: WASI call failed with BADF
WASI error: WasiError: WASI call failed with BADF
WASI error: WasiError: WASI call failed with BADF
]]></system-out>
  <system-err><![CDATA[]]></system-err>
</testsuite>

Is it expected?

[Svyatoslav Kuzmich [JB]]

Yes, add you directory into preload and you will see it in fd 4!

[sdeleuze]

Also is there a way to have autocomplete in IDEA with kotlin.wasm.unsafe. API?

[sdeleuze]

Not a big deal if not possible

[sdeleuze]

But asking in case of

[Svyatoslav Kuzmich [JB]]

Known problem, its fixed internally in IDE plugin and will probably be shipped next release

[sdeleuze]

Reading https://github.com/JetBrains/kotlin-native/issues/1185#issuecomment-718721267 I guess there is no hope to use to use UTF-8 for internal encoding of string in Kotlin/Wasm, so interop with Wasm Components will likely be costly (no way to just pass string args/return values) right?

[Svyatoslav Kuzmich [JB]]

Very unlikely that Kotlin with switch to UTF-8 soon.
However, Component Model strings are abstract sequences of Unicode scalar values. Canonical ABI supports UTF-8, UTF-16 and latin1+utf16 encoding variants. It seems that conversion will happen only if different sides of interface have different supported string encodings.
Even if encodings are incompatible, there are different techniques to reduce overhead. We could do lazy conversion only when we actually access characters by index. We could also do a technique similar to Swift’s breadcrumbs, where we could keep source encoding without copy but allow somewhat costly, but amortised O(1) access to UTF-16 characters.
There is also a stringref proposal, an abstraction over these multiple-encodings-at-once approach that could become a standard.

[Svyatoslav Kuzmich [JB]]

kotlin.wasm.unsafe.* and WasmImport are now merged into master with a few changes. Allocator scopes can now be nested. And Pointer is now a value class with load and store methods.
Gradle configuration of nodeJsArgs used in my experiments is not merged yet.

[Svyatoslav Kuzmich [JB]]

Feedback on new APIs is welcome as usual :)

[sdeleuze]

Nice, I will switch on master then, that will be easier. I had less time than expected on my computer, but will provide feedback as I progress on my experiments.

[sdeleuze]

For now I am spending time to fully understand what they did on TeaVM Wasi fork, before starting fully on Kotlin/Wasm side.

[sdeleuze]

Hey, happy 2023! I am having a deeper look to kotlin.wasm.unsafe.* API, Kotlin/Native cinterop, etc.

Current API provide access to individual Byte to Long values, do you have already in mind what could be possible in term of exposing byte arrays access. Does it make sense to expose an API hat allow to store/load byte arrays from a Pointer + a size ? Would that involve copying those data or can we achieve zero copy interop? What types would be involved (ByteArray, something else)?

[Svyatoslav Kuzmich [JB]]

It would make sense to add load and store functions for ByteArray some time in the future, yes. Right now we don’t need any compiler support for this, as they will be implemented using a loop of individual loads/stores. But in the future Wasm GC proposal would likely add instructions to copy memory in bulk, in an optimised way.

We do need a copy now because Wasm GC arrays and Wasm linear memory are separate by design. We will avoid the extra copy in the future when Component Model will add Wasm GC support to its ABI

[sdeleuze]

Not directly related to the new API, but I have a few other open questions.

[sdeleuze]

If at some point Kolin/Wasm supports WASI/WIT, would we have potentially a kotlin { wasm { wasi() } } Gradle configuration? If that's the case, println() and similar methods could just use WASI capability to print to standard output?

[sdeleuze]

I think my main point of concern is the lack of Kotlin multiplatform API for let say file IO and other scope covered by WASI.

[sdeleuze]

For files, I guess https://square.github.io/okio/multiplatform/ or https://docs.korge.org/korio/ could be updated to support WASI.

[sdeleuze]

But the lack of real standard library outside JVM is a concern.

[sdeleuze]

Any plans on that or do you rely on the community to move forward?

[sdeleuze]

I understand the potential pain/cost in term of maintenance.

[sdeleuze]

But lack of Multiplatform library cause a lot of fragmentation and prevent reuse.

[sdeleuze]

Or should we start something more specific to WASI? Not sur.

[sdeleuze]

Any thoughts?

[Svyatoslav Kuzmich [JB]]

If at some point Kolin/Wasm supports WASI/WIT, would we have potentially a kotlin { wasm { wasi() } } Gradle configuration? If that’s the case, println() and similar methods could just use WASI capability to print to standard output?

I think so. JS and WASI seems like a two leading “system interfaces” for Wasm that should warrant first-party support.

[Svyatoslav Kuzmich [JB]]

We would want to encourage multi-platform libraries to support Wasm target(s) when it is ready. It looks like the issues with libraries fragmentation are not unique for Wasm, they are shared with more mature platforms like K/JS and K/Native, and these platforms might be a better place to solve them right now. Making a first-party library with JVM/JS/Native support and then porting it to Wasm would probably be easier to justify than the other way around.

[sdeleuze]

Yeah indeed, nothing specific to Kotlin/Wasm.I may start a related discussion in #multiplatform.

[sdeleuze]

I will continue experimenting with the new API once nodeJsArgs is merged.

[Svyatoslav Kuzmich [JB]]

nodeJsArgs support is merged.

Repo https://github.com/skuzmich/kotlin-wasi-bindings-experiments is now updated to be compatible with master.

[sdeleuze]

I guess you already know, but Swift switch from UTF-16 to UTF-8 and this article provides interesting background. https://www.swift.org/blog/utf8-string/

[sdeleuze]

I saw also a reference to breadcrumbs you mention earlier

[Svyatoslav Kuzmich [JB]]

If Java switches to UTF-8, Kotlin will most likely follow suit. 😉

[sdeleuze]

Yeah I know Kotlin has other interop constraints ^^

[Svyatoslav Kuzmich [JB]]

APIs too. kotlin.String has a many methods that operate on UTF-16 indices. They are not hidden away under some UTF16View