https://kotlinlang.org logo
Join Slack
Powered by
So, while <indexeddb WASM support will take a whil...
# webassembly
s
So, while indexeddb WASM support will take a while, what are the other currently available options to store/cache a 10 MB file in the browser? I asked ChatGPT & Gemini to generate me some code to use IndexDB, but that all failed. Are there other libraries out there that I missed?
👀 1
✅ 1
r
I think 
kotlin-browser
from the wrappers project has bindings for IndexedDb. But I haven't used it.
👍 1
👍🏾 1
s
Interesting. It's so new ML doesn't know it.
e
That might make the JuulLabs implementation much easier (sorry about the very long delay there 😬), since it currently has to implement the IndexedDb spec, and the Kotlinified layer on top of it.
s
You mentioned something like that in the linked thread, but I’m not familiar with all the details.
I wonder why JuulLabs is not interested in pushing to WASM.
e
I was referring to using the feature that kotlin-browser used to commonize between js and wasm-js, but if they wrote the bindings already I'd be very happy to use them instead of maintaining a separate set 😅 I have no affiliation with them so I can't say, but I'd imagine it's because they don't use it, so it's more of a community nice to have (which is a trend I've noticed in lots of Android OSS lately for better or worse)
👀 1
👍 1
w
What are you trying to store? Could you just use the browser Cache APIs to do it? Or do you actually need to structure the data? Not sure if the 10MB you're talking about is the data size or you literally have a 10MB file to store.
s
My main goal is indeed caching. I create search indexes for my Oxygen Not Included Seed Browser. I splitted them in smaller files. Each 5 to 10 MB, everything together maybe 100 MB. You think there are other appropriate ways to prevent downloading these files over and over again?
Those are protobuf compressed files. I don’t need any structure, just key/value
w
I'm not an experienced webdev, but I think you can just make a Cache object and stuff it with the response: https://developer.mozilla.org/en-US/docs/Web/API/Cache/put And just read from the cache first when you need the file. You can use any request key you want for each value.
This is how I do offline-first for my web app/PWA. Although in my case I force a service worker to always have all files available offline first before letting the app launch/worker take over.
s
Service worker to have everything offline available sounds good. I try to make https://stefan-oltmann.de/oni-seed-browser/ more offline-first. The map database is 5 GB, but that’s now distributed via S3 and so it will be less depend on a single server. Just the search is a online feature and I want to do that offline by downloading a protonuf compressed version of the database.
Does this Web Cache API have proper Kotlin binding or a lib to use them? I don’t like that JSExports stuff.
Kotlin Multiplatform Settings has just local storage… and that has a 5 MB limit unfortunately.
r
Kotlin wrappers to the rescue once again :-)
👀 1
✅ 1
s
Ok, I guess I really need to check those out 😄
t
We also also support workers/shared workers/lazy functions creation without separate subroject (via plugin) ;)
😮 1
m
As far as I understand it this cache API is only available in a web worker, which means you have to write the web worker code in Kotlin too which I have avoided so far because it looks not so easy.
t
Service worker - separate use case, but we are open for discussion and implementation.
w
m
@turansky That sample app looks interesting. I’ll have to have a look at it tomorrow.
😜 1
@Winson Chiu Maybe you are right. The docs say: “This feature is available in Web Workers.” which does not exclude the possibility that it is also available from the main program.
r
t
👍 2
k
kotlin-lmdb supports wasm among others, I use it for exactly that, a serialized state and request caching https://github.com/crowded-libs/kotlin-lmdb
s
I'm not sure how this helps me to store & load data in the browser.
r
Probably works, but using an external database instead of something built-in in the browsers seems too heavy for me.
☝️ 1
➕ 1
k
put("file".tobytes(), fileBytes) tx.commit()
It's a 500 line c file, and a tiny wrapper on top. I'm not sure why that's "heavy". Everything uses IndexDB under the covers for the filesystem (that's how emscripten built their virtual fs).
s
To me it sounds like I can save a file within LDAP using that lib. I want to save something inside a default browser; something that works on every platform and browser combination.
Everything uses IndexDB under the covers
That confuses me.
m
@katokay To me this looks awesome! I have to give it a try.
r
This lib seems to give you nice api by using some additional layers. Probably not a bad idea after all 🙂
s
I'm lost. What has LDAP to do with IndexedDB?
r
Not LDAP. LDAP's LMDB database.
s
Do they use the same architecture?
r
Kotlin lib -> LMDB compiled to wasm -> emscripten -> virtual fs -> indexedb. Am I correct @katokay?
👀 1
m
Emscripten emulates a file system on the browser via IndexedDB. Maybe this helps: https://emscripten.org/docs/porting/files/file_systems_overview.html This is something that I am missing in wasmJs.
👀 1
k
Yes, you are correct @Robert Jaros
lmdb is the database used in OpenLDAP under the covers, it was created by the company behind openldap symas. the key/value store has nothing to do with ldap itself
💡 1
👍 1
m
Its a Kotlin multiplatform wrapper arround https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database, right?
k
Yes @Michael Paus and if you're curious. https://github.com/crowded-libs/kotlin-lmdb/blob/main/build.gradle.kts#L378 is where it informs emscripten to include filesystem and memory management utilities in the js file used on wasmJs native-interop
🙏 2
K 2
m
This sentence in the README puzzles me a bit. You will need to setup a copy task for the js assets with your own project to ensure the modules load successfully. I haven’t figured out how to make the packaging work seamlessly for consumption of the maven package yet. What exactly has to be copied and where does it have to be copied to?
k
https://github.com/crowded-libs/duks-storage-lmdb shows an example of what that means
It's been very frustrating for me not to figure out the right way to package the wasm interop files in a way that is maven package consumption friendly. I've posted questions but haven't seen any recommendations that works.
r
I would probably do this with a gradle plugin.
I think compose gradle plugin does something like that to skiko.wasm.
k
I agree and had considered it. Just feels overkill for a small library and something the larger ecosystem should already be addressing without plugins. Also, couldn't help that nagging feeling like I just hadn't figured out the right way to do it to make it work.
t
In common case it's responsibility of bundler. Responsibility of libraries and Kotlin compiler - create fine URLs.
m
The examples in the README of kotlin-lmdb don’t seem to be compatible with the latest version 0.3.4 of the lib. Also, what is the meaning of “/path/to/database”? Whatever I put in there I only get a 
lmdb.LmdbException: No such file or directory
(Tried it first on JVM.)
k
The directory has to exist first. It's the directory the database file will be stored. Only exception to that is on WebAssembly because of io not working on wasmJs targets needs to create the directory itself.
m
OK, got that working now. I also had to add 
.toVal()
in several places. In case of WebAssembly. What are good values for 
WasmUtils.mountBestFilesystem("???")
and then 
env.open("???")
?
k
WasmUtils.mountBestFilesystem("/tmp"), then yes you can open the environment afterwards.
I tried to move that inside the open implementation for webassembly, but I can't remember why it didn't work, I think it might have been a race condition for when assets were available, but it's worth me looking at again. I don't like that it needs the extra steps. This particular target took a lot of trial and error to finally get an end-to-end working codebase with passing tests. I took like 5 different approaches until I finally found the path that worked.
m
I am still not sure what to copy. Do you mean the 3 items (“lmdb-wrapper.mjs”, “lmdb.mjs”, “lmdb.wasm”) from 
kotlin-lmdb/src/wasmJsMain/resources/kotlin/
or the ones from 
kotlin-lmdb/src/wasmJsMain/resources/
or something else? The variants don’t seem to be exactly the same.
k
That's correct, those are the files needed.
You can also manually copy them into your local wasmJs resources directory manually just to try out and it will load them successfully from there.
m
Yes, but which set? The one from the kotlin sub-folder or the other one. The files in there differ.
k
I believe it's the ones in the kotlin folder. FWIW, I tried the resources directory in the package and that didn't change anything. I think it only works for kotlin-native supported targets which wasmJs isn't one to my knowledge.
r
My way of handling such resources in js/wasmjs targets is to publish them as standard NPM package to npmjs.com (you can use #C01LQTET9P1 plugin for that). When the package is published you can use it as npm dependency in the library. It will make the package automatically downloaded and available in the consumer projects. You can just replace 
@JsModule("./lmdb-wrapper.mjs")
with 
@JsModule("my-npm-package/lmdb-wrapper.mjs")
and it should work.
k
That will likely be the route I take. I hate it at the same time. Just feels like it should be a first-class supported scenario shipping resources for a wasmJs package. Thanks for recommending.
KT-80547 kotlin-native for wasmJs vote it up if you agree
thank you color 2
m
Just voted for this issue because I am facing the same problems for a long time now. However, I fear the answer will be to wait for the component model which is supposed to address many of these things. I am wondering though when the component model will finally see the light of day in the Kotlin ecosystem. According to the latest road map all activities in that direction have been postponed. See, e.g.: https://youtrack.jetbrains.com/issue/KT-64569/Kotlin-Wasm-Support-Component-Model
k
Agreed, and I'm simplifying things, it seems like a lot of the plumbing should already be there to support this request with fairly minimal effort.
m
I hope so 🙏.
I just hacked together a little key-value store for wasmJs via the Cache API and by using kotlin-wrappers its actually quite simple even if you want to store binary data and not just strings. It works and I can now store large amounts of binary data in my PWA 😆. File MonsterStore.kt in commonMain:
Copy code
package de.mpmediasoft.webcache.webcachedemo

expect class MonsterStore {

    suspend fun open(name: String)

    suspend fun store(key: String, data: ByteArray)

    suspend fun retrieve(key: String): ByteArray?

    suspend fun delete(key: String)

    suspend fun dispose()

}

expect val store: MonsterStore
File MonsterStore.wasmJs.kt in wasmJsMain:
Copy code
package de.mpmediasoft.webcache.webcachedemo

import js.buffer.toArrayBuffer
import js.typedarrays.toByteArray
import web.cache.*
import web.http.BodyInit
import web.http.Response
import web.http.bytes

actual class MonsterStore {

    private lateinit var name: String
    private lateinit var cache: Cache

    actual suspend fun open(name: String) {
        this.name  = name
        this.cache = caches.open(name)
    }

    actual suspend fun store(key: String, data: ByteArray) {
        cache.put(key, Response(BodyInit(data.toArrayBuffer())))
    }

    actual suspend fun retrieve(key: String): ByteArray? {
        return cache.match(key)?.bytes()?.toByteArray()
    }

    actual suspend fun delete(key: String) {
        cache.delete(key)
    }

    actual suspend fun dispose() {
        caches.delete(name)
    }

}

actual val store: MonsterStore = MonsterStore()
Use these coordinates for kotlin-wrapper:
Copy code
kotlin-browser = "2025.8.21"
kotlin-browser = { module = "org.jetbrains.kotlin-wrappers:kotlin-browser", version.ref = "kotlin-browser" }
🔥 1
thank you color 2
👍 1
👏 2
s
That's awesome. Thanks.
Interestingly no AI (not ChatGPT, Gemini, Grok or Claude) was able to write that code. They don't seem to have kotlin-browser in their training data.
The kotlin-wrappers are nice. I need to explore them more. 🙂
k
@Stefan Oltmann it's helpful if you use context7 mcp, even if it doesn't exist already you can point context7 to a github repo and it will index the samples from the readme. It helps a lot.
💡 1
s
Thanks for the idea. That's another thing I need to explore. So much to learn. 😄
t
@Michael Paus
Copy code
// before
cache.match(key)?.bytes()?.toByteArray()

// after
cache.match(key)?.byteArray()
?
m
Yes please!
t
It should work right now
m
Not in the version that I am using right now.
t
You requested 
byteArray()
previously - we added it 😉
🎉 1
m
Where have you hidden this? IntelliJ is not offering me any import for that.
➕ 1
👀 1
@turansky You are probably referring to this but I can’t find the equivalent for a Response.
t
It was error in extension declaration 😞
Will be available in next release 😉
👍 2
Kotlin Wrappers 
2025.8.22
released. Will be available on MC in few hours 😉
K 2
m
Why 22? It’s the 29th already today 😉.
t
It's 22nd release in August
💡 2
And we count from 
0
😉
23rd release in fact
s
Oh wow, I also thought this is for the date. 😄
t
Today we did 2 releases 😉
👀 1
m
Hmm, it’s not yet available on MC even after 15 hours. Maybe it’s not allowed to do more than one release per day 😢.
t
Just 1 missed click ;)
❓ 1
Button pressed. Release is ready!
🎉 2
m
Just converted my PWA to use a web cache base file vault implementation instead of the old limited local storage based one 🎉.
s
Did the same. Works great. 🙂
It looks like, despite its name, local storage is really just intended for storing app settings.
m
Isn’t it a good feeling to still be superior to the KI 😉?
💯 1
s
At least here. AI can write my JavaScript slop for the Cloudflare workers that don't look so nice with Kotlin/JS 😛
I'd buy some Cloudflare stocks if they made the workers run native Kotlin code. Given their free tier and ease of use compared to AWS services that would be a game changer.
12 Views
Open in Slack
PreviousNext
Powered by