Will there ever be an easy way to handle `Map<S...

# serialization

[Richard Gomez]

Will there ever be an easy way to handle Map<String, Any>? It's a legitimate use case that seems to have been dismissed outright.

Saying "use JsonObject instead" but not providing any guidance is a recipe for frustration and buggy implementations:
https://github.com/Kotlin/kotlinx.serialization/issues/296
https://github.com/Kotlin/kotlinx.serialization/issues/746
https://github.com/Kotlin/kotlinx.serialization/issues/1537
https://kotlinlang.slack.com/archives/C7A1U5PTM/p1632495920135600?thread_ts=1632392673.125200&cid=C7A1U5PTM

[ephemient]

there will never be a general way to handle serializing Any , safely - Jackson/Gson aren't safe, in that they can fail at runtime rather than compile time

[ephemient]

for the same reason that kotlinx.serialization doesn't support open polymorphism without manual work, you'll need to put the work in to determine which Any types you actually want to handle

Untitled.cpp

[Richard Gomez]

there will never be a general way to handle serializing Any , safely - Jackson/Gson aren't safe, in that they can fail at runtime rather than compile time

While I understand the rationale, saying "just use JsonObject" is a bit of a cop-out. Requiring people to write their own serializers doesn't actually absolve kotlinx.serialization of the issue — those user-created serializers can also fail at runtime. Is people writing custom serializers that rely on reflection really better than something like Map<String, @Contextual Any?> that fails if it can't find a serializer at runtime?

It just feels like clash between correctness and pragmatism.

[Richard Gomez]

Don't get me wrong, I like kotlinx.coroutines, but there are enough of these types of issues which make it difficult/impossible to use instead of Jackson, Moshi, etc.

[ephemient]

like Any? anywhere else, it's dangerous (security-wise) to allow uncontrolled input to deserialize to and instantiate arbitrary classes

[ephemient]

https://rules.sonarsource.com/java/tag/owasp/RSPEC-4544 in Jackson, for example. it's easy but it shouldn't be

[Richard Gomez]

As an aside, I'm also not a fan of using kotlinx.serialization.json.JsonObject in application code. The implementation is technically correct, but difficult to use compared to javax.json.JsonObject or Vert.x's JsonObjec

e.g. https://kotlinlang.slack.com/archives/C7A1U5PTM/p1633134038183200

[ephemient]

JSON on its own doesn't have any typing ahead of time, so it'll depend on your application whether you want to treat unknowns as Double (like Javascript does), optimistically try to stuff things in Int/Long/BigInteger, downcast to smaller integral sizes, allow arbitrary precision with BigDecimal, etc.

[ephemient]

also kotlinx.serialization works with other formats such as Protobuf, where there's no natural way to represent Map<String, Any?> anyway

[Richard Gomez]

like Any?  anywhere else,  it's dangerous (security-wise) to allow uncontrolled input to deserialize to and instantiate arbitrary classes

@ephemient: I agree; I guess my point is that people are going to do it anyway if there's a need, and in my experience it's easier to do it the unsafe way (custom serializer with reflection, or use Jackson) than the safe way. There may already be a really simple and safe way to handle Map<String, Any>, but the discussions and documentation tend to leave the details as an exercise to the reader.

[Richard Gomez]

Plus I don't think that the average person has enough patience to read through the entire documentation, experiment with serializers, and write a bunch of custom code for each model in their application. They just want to serialize & deserialize their models. So instead of nudging people towards an objectively better approach (safer, faster, compile-time errors), they'll just go back to Jackson.

[ephemient]

I think there's a bunch of things a Map<String, Any> values could be, either JsonElement-like primitives (but numeric primitives can't round-trip, which all default serializers try to do, and it's application specific what you want them to deserialize to), or polymorphism including some subset of program types, or open polymorphism. none of this can be encoded statically in the types and so there isn't a clean way to express it in kotlinx.serialization either

[ephemient]

kotlinx.serialization makes a type-safe cross-platform cross-format subset of operations easy, and this isn't one of those

[Richard Gomez]

I think there's a bunch of things a Map<String, Any> values could be... so there isn't a clean way to express

Again, I understand why they don't do certain things. But I think it'd be useful to have implementations or well documented examples for people who need to do them.

As with global custom serializers or serializers for types like UUID or Java Time, the philosophy seems to be "this could be used differently 1% of the time, so we won't provide help for the other 99%". To steal a point from @thanksforallthefish: kotlin serialization often feels overly generic, which makes it tedious to use compared to something like Jackson.

[Richard Gomez]

I'm saying this as someone who's used it for quite a while now, and has gone through the effort of replacing Jackson where possible.

[Richard Gomez]

I just wish that Kotlin Serialization was as powerful, intuitive, and easy to use compared to its alternatives, as Kotlin itself. 🙂

[ephemient]

concerning Java and JSON only, then kotlinx.serialization is overly-generic, I agree. but that is an area that is already handled by other libraries; kotlinx.serialization fills a role that nothing else satisfies

[ephemient]

and the API in general is curated to the subset of things that can be done reasonably safely and efficiently, just like Kotlin itself generally limits its features to things that can be implemented reasonably and efficiently (as opposed to Scala's grab-bag of features, many of which have surprising interop or runtime costs)

[Big Chungus]

You could try Kon. It has two-way conversions between such map and JsonObject

[ephemient]

if the Any types are all primitives (and Int/Double suffice) or List/Map, yes. but upthread, OP seemed to want some sort of serializer lookup, which is definitely not going to be supported out of the box

[ephemient]

there's going to be "simple" lists and maps that can't be converted to JsonElement too, for example

buildList<Any> { add(this) }
buildMap<String, Any> { put("this", this) }

are legal Kotlin objects but any .toJsonElement() will fail

[Big Chungus]

Correct, but that couldn't be represented in valid json anyways

[ephemient]

it's not possible (with public methods) to construct a self-referential JsonElement so I'm just saying there are reasons to make that the public interface, not Any