Hello good people. I'm literally losing my mind on...

# serialization

[Davide Giuseppe Farella]

Hello good people.
I'm literally losing my mind on this 😅

We have a data structure that has a JsonElement and an integration test that verifies that said model is equal to another one.
The test fails because a model as a 0.000001 in a value of the Json, while the other has 1E-5.
I spent a ridicolous amount of time on this and I even discovered Kotlin automatically convert to scientific notation when calling toString() (screen #1)

I've been checking JsonElement source and I'm wondering how could I even end up with a JsonPrimitive, that is not a string, but has the decimal representation? (screen #2) 😅

[ephemient]

something like this? https://pl.kotl.in/fpdzB1fSX

[Davide Giuseppe Farella]

Mh, indeed we decode to JsonElement, but I still cannot wrap my head around it 😅 why it doesn’t convert to scientific notation?

[ephemient]

on the first part, kotlinx.serialization.json preserves the formatting of the original source document, because it doesn't know what type of number will want out of it. perhaps it will end up being decoded to a BigDecimal, or perhaps not.
on the second part, kotlinx.serialization.json (and Kotlin in general) doesn't implement its own number formatting. it simply uses the platform number stringification, which in the case of JVM, can use scientific notation, which is permitted by JSON

[Davide Giuseppe Farella]

Oh, thank you 🙏
Any way to deal with my issue, not being aware of what the json contains?

[ephemient]

it depends on context whether two JSON blobs represent the same value or not. if you only have the JsonElement then you've lost the context

[ephemient]

{ "value": 0.33333333 }, { "value": 0.3333333333 }

if you're decoding them into a

@Serializable data class Struct(val value: Float)

then those two are equal. if you're decoding them into a

@Serializable data class Struct(val value: Double)

then those two are not equal

[Davide Giuseppe Farella]

Thank you, we’re decoding into Struct(JsonElement)

[Davide Giuseppe Farella]

Our case is something like this, and indeed the format is lost 😕~https://pl.kotl.in/ReYrU0cCK~

[Davide Giuseppe Farella]

Ops, it didn't update the code 😄

[Davide Giuseppe Farella]

https://pl.kotl.in/oCWHYgtM9

[ephemient]

yeah… you can't expect exact round-trip behavior in kxs or other json implementations either

[Davide Giuseppe Farella]

Gotcha 🙏
Any way to keep it 1:1? We don’t really care how we store it, we’re using JsonElement because:
1. It ensures the string we receive from API is a valid JSON at the root
2. We can easily serialize it and store it

[athos]

on the first part, kotlinx.serialization.json preserves the formatting of the original source document, because it doesn't know what type of number will want out of it. perhaps it will end up being decoded to a BigDecimal, or perhaps not.

This makes sense, but then why when encoding the object to a json string, it treats the test JsonPrimitive field as a number instead? Wouldn't it be more appropriate to serialise it as a primitive?

My expectation is that until someone somewhere explicitly needs to try and turn the primitive into a number, that shouldn't happen, and more so if it's just to "write" that primitive into a json string.

[ephemient]

Any way to keep it 1:1?

fun JsonElement.preserveLiterals(): JsonElement = when (this) {
    is JsonArray -> buildJsonArray {
        for (item in this@preserveLiterals) add(item.preserveLiterals())
    }
    is JsonObject -> buildJsonObject {
        for ((key, value) in this@preserveLiterals) put(key, value.preserveLiterals())
    }
    JsonNull -> JsonNull
    is JsonPrimitive -> if (isString) this else JsonUnquotedLiteral(content)
}

why when encoding the object to a json string, it treats the test JsonPrimitive field as a number instead?

when encoding JsonElement picks some numeric type, and kxs guesses Long, ULong, or Double depending on the content

[Davide Giuseppe Farella]

Thank you, sadly this might be infeasible, as it would have a significant performance penalty in our context.
I was hoping in some config or any trick to overcome this.

As now the only solution is to have a custom asserter that navigates both the container and the JSON fields and, when encountered a Primitive, try to parse its content to a Number.
This is inconvenient, as we would prefer to avoid mutation at least when serializing the JsonElement, and do not resort to complex assertions.

[athos]

when encoding JsonElement picks some numeric type, and kxs guesses Long, ULong, or Double depending on the content

I think it would be more consistent if it didn't do that. The "type" of the field in JsonElement should be handled agnostically, as a "literal", in both directions (decoding, which is already the case, and encoding). It should only be interpreted when/if someone needs to take that value outside of JsonElement .