Hey. I need to expose some JS Stream API for k/js ...

# webassembly

[Alexander Zhirkevich]

Hey. I need to expose some JS Stream API for k/js and k/wasm. I've easily done it for k/js, but struggling on how to do it for k/wasm 🧵

[Alexander Zhirkevich]

Basically what i need to do is to create ReadableStream from kotlin ByteArray.

ReadableStream constructor takes an object that has a start method.

For k/JS i have

external interface UnderlyingSource {
    fun start(controller: SourceController)
}

external interface SourceController {
    fun enqueue(data : ByteArray)

    fun close()
}

external class ReadableStream(
    source: UnderlyingSource
)

And then i create anonymous object and pass it to the constructor

val source = object : UnderlyingSource {
    override fun start(controller: SourceController) {
        controller.enqueue(array);
        controller.close();
    }
}

val stream = ReadableStream(source)

Everything works fine.

How to achive the same result in k/wasm?

I tried the following to create the underlyingSource and used ByteArray.toJsReference() as a parameter but with no luck. JS says that data is not an array buffer

fun createSource(data : JsAny) : JsAny = js("""
({
   start(c){ 
     console.log(data)
     c.enqueue(data)
     c.close()
   }
})
""")

[Svyatoslav Kuzmich [JB]]

To convert Kotlin array into JS array, you can use the Int8Array type and manually copy the elements like this.
References created by .toJsReference() are not accessible in JS; they are only useful when they are sent back to Kotlin.

[Alexander Zhirkevich]

Thanks! it works!
Is it the most performant way to pass arrays between k/wasm and js? The array can be up to several megabytes in size and (probably) can cause a UI junk. Don't really want to spawn a web worker for this

[Svyatoslav Kuzmich [JB]]

Current Wasm GC limitations force you to do a per-element copy, but it is probably not the fastest, because it does Wasm->JS call for each element.
A few of ideas for speeding it up:
• reduce the number of Wasm->JS calls by using intermediate linear memory (an example of the opposite ArrayBuffer.toByteArray operation).
• do the loop with element copy on the JS side by calling small exported wasm functions, as small JS->Wasm calls could be inlined by browsers.

[Svyatoslav Kuzmich [JB]]

You can try if this would be faster:

import org.khronos.webgl.Int8Array

@JsExport
fun kotlinArrayGet(a: JsReference<ByteArray>, i: Int): Byte = a.get()[i]

@JsExport
fun kotlinArraySize(a: JsReference<ByteArray>): Int = a.get().size

fun byteArrayToInt8ArrayImpl(a: JsReference<ByteArray>): Int8Array = js("""{
  const size = wasmExports.kotlinArraySize(a);
  const result = new Int8Array(size);
  for (let i = 0; i < size; i++) {
     result[i] = wasmExports.kotlinArrayGet(a, i);
  }
  return result;
}""") 

fun ByteArray.toJs(): Int8Array = byteArrayToInt8ArrayImpl(this.toJsReference())

You can also remove this extra copy if you can manage to use exported kotlinArrayGet and kotlinArraySize in your JS directly, or with a wrapper layer. Especially great when only a portion of ByteArray is accessed in JS.

[Alexander Zhirkevich]

The main goal actually is to decompress deflated zip archive stored in ByteArray using browser DecompressionStream api

[Svyatoslav Kuzmich [JB]]

Could splitting big array into smaller chunks and processing each chunk async help?

[Alexander Zhirkevich]

I split it by zip entries. And compression stream uses 65k bytes chunks under the hood. As far as I know compression is performed in the bg thread, so may be I am able to do conversion stuff on that thread too

[Alexander Zhirkevich]

Isn't wasmExports stuff internal? Can i use it for a public library?

[Svyatoslav Kuzmich [JB]]

It is a temporary solution, we may replace it with something better later.

[Svyatoslav Kuzmich [JB]]

This is one of the feature that might require your users to update your lib version to be compatible with the latest compiler. And this is also true, to some degree, for other Kotlin/Wasm-specific features at this stage.