how do you stream serialize/deserialize an array o...

# serialization

[Bernhard]

how do you stream serialize/deserialize an array of objects? think of gigabytes of JSON

[Casey Brooks]

Kotlinx Serialization does offer streaming APIs, but they're only available on JVM. And these may still be tricky to use as the full source object is still likely in memory, even if the JSON parsing/transformations are streamed.

https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-json/kotlinx.serialization.json/decode-from-stream.html

https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-json/kotlinx.serialization.json/decode-from-stream.html

You'll probably find it easier to pass the text stream directly as JSONL and only use Kotlinx Serialization on each individual row. So the library itself is not streaming the large objects, you are. But you can still deal with each object safely.

https://jsonltools.com/jsonl-vs-json

[Bernhard]

I found decodeFromStream and am on the JVM, but I haven't found any examples on how to use those APIs

[Casey Brooks]

Basically the same as using it without streaming. You just pass in an InputStream instead of a String.

@Serializable
data class User(val name: String, val age: Int)

val text = """{"name": "Alice", "age": 30}"""
val stream = text.byteInputStream()
val user = Json.decodeFromStream<User>(stream)

But again, this mostly just operates at the byte-decoding level, and doesn't really work for processing the entire source in smaller chunks, since it still collects the results in memory and returns the whole thing when its done. It just helps avoid doubling the memory by not holding both the input text AND the decoded objects in memory.


A real streaming approach would be finding some way to not need an entire JSON Array at once, but to process the small chunks individually. Again, the core Kotlinx.Serialization isn't really built to handle this itself, but it's pretty each to build your own streaming pipeline:

val jsonl = """
    {"name": "Alice", "age": 30}
    {"name": "Bob", "age": 25}
    {"name": "Carol", "age": 35}
""".trimIndent()

jsonl.lineSequence() // could be any method of reading content line-by-line. Reading a file, receiving data from a socket, etc.
  .map { Json.decodeFromString<User>(it) }
  .forEach { user ->
    println(user.name)
  }

[hfhbd]

You can also use the decodeToSequence overload to basically support JSONL or an array wrapped response: https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-json/kotlinx.serialization.json/-decode-sequence-mode/

[hfhbd]

But as soon as you have a nested array with many elements, it will be fully loaded into the memory.

[Casey Brooks]

I stand corrected, thank you for pointing that out @hfhbd! i was not aware of that API as it looks like it's also JVM-only, and I work almost exclusively with KMP

[hfhbd]

The io method isn't JVM only: https://github.com/Kotlin/kotlinx.serialization/blob/master/formats/json-io/commonMain/src/kotlinx/serialization/json/io/IoStreams.kt#L97

[Bernhard]

yeah, what I need is to parse some wrapper object which holds a huge list of elements inside

[Bernhard]

in jackson I used a custom json parser for that that looped over the token stream until it found the inner list, then serialized each element in a loop and wrote it into the db, looks like I won't be able to do that in Kotlin in that case

[Juan Hurtado]

If you are open to using other libraries, you can do this token-by-token streaming in Kotlin with Ghost. Since GhostJsonReader is public, you can do a manual token-by-token loop:

val reader = GhostJsonReader(bufferedSource)
reader.beginObject()
// ... read keys, then:
reader.beginArray()
while (reader.hasNext()) {
    val user = userSerializer.deserialize(reader) 
    // O(1) memory
}

Ghost Android Guide — Flat vs Streaming Section
Ghost Architecture — Reader Pipeline
If you want to try it out 🙂

[pdvrieze]

The challenge with "streaming" is how to provide the result of incremental parsing in a dynamic way. There is no straightforward way to do so. The only thing that appears to work is effectively deserializing a collection/list sequentially, but such lazy list can generally not be stored in an outer object (as it needs other bits to be parsed).

Good formats (including Json/XML support nested parsing so it is possible to parse the outer container regularly, and use serialization for the individual parts)

[Bernhard]

I'm doing exactly that with Jackson right now

[Bernhard]

question I'm pondering right now if I can replace jackson with kotlin serialization but the lack of streaming parsing/serialization kinda kills it

[pdvrieze]

@Bernhard The base parser for Json supports streaming parsing. It is deserialization that works such that it needs to fully parse the data before it returns. This means that if you parse the full document directly it reads the full document immediately. However, you can use serialization to partially parse fragments that are worthwhile units of incremental parsing. You can then incrementally process those deserialized objects withouth having to first parse the entire document.

[Bernhard]

where can I find the docs on that?

[pdvrieze]

@Bernhard I had a look. The lexer/parsing classes used by the Json format are internal. They are also fairly tightly integrated, so making them public would probably not work. (I maintain xmlutil that supports XML - it actually has a the serialization code work on top of a regular pull parsing code).

[Bernhard]

I'm familiar with baeldung.com/jackson-streaming-api

[pdvrieze]

@Bernhard There is possibly a case for having the official Json format support being created on top of a general Json parser (and generator). This would need to be multiplatform but would be feasible. You should probably create an issue for it (with use case). But the gigabytes size makes clear that (de)serialization of parts of a json document would be needed somehow (the container using direct parsing, serialization for the elements).