https://kotlinlang.org logo
#serialization
Title
# serialization
v

Vasileios Chroniadis

01/31/2023, 9:24 PM
If a field is added in a JSON response, is this considered an API change? Yes 👍 / No 👎 ?
👍🏾 1
👍 5
👎 1
e.g. from
Copy code
{
  "name": "John",
  "age": 30
}
to
Copy code
{
  "name": "John",
  "age": 30,
  "car": null
}
k

kevin.cianfarini

01/31/2023, 9:29 PM
yes. It’s a backwards compatible API change.
c

Casey Brooks

01/31/2023, 9:30 PM
Yes, this would typically constitute a breaking change to the API. Even if the value of
"car"
is
null
, if a consumer has as class which does not have a
car
property and
ignoreUnknownKeys
is
false
, the new JSON response would throw an error
In terms of semantic versioning, this would be a minor change. An older client cannot be expected to work with the new value, but newer clients should be able to work with older responses if
car
is nullable
k

kevin.cianfarini

01/31/2023, 9:33 PM
Yeah I was also wondering how
ignoreUnkownKeys
would play into this. I think for most other deserializing clients (eg. the general question) this would be backwards compatible.
e

ephemient

01/31/2023, 9:54 PM
it depends on the expectations between the endpoints involved, but this is typically considered to be a compatible change.
v

Vasileios Chroniadis

01/31/2023, 9:59 PM
If ignoreUnkownKeys default is false, which will crash the deserialiser with such a change, then how is that typically considered to be a compatible change?
k

kevin.cianfarini

01/31/2023, 10:04 PM
kotlinx serialization is the first deserialization client I’ve ever used that has this “strict” mode enabled by default.
For the most part additions to a payload are considered backwards compatible (aside from here, where the default is to throw on unknown keys)
c

Casey Brooks

01/31/2023, 10:10 PM
It might not be the “standard” to be so strict, but consider that, unlike most serialization libraries, this one is designed so that you use the same serialization models on both the server and client. Most other languages that can do that are either dynamic and don’t really need specialized serialization libraries (JS), or else use different languages/libraries between client and server and can’t make any assumptions about the structure of JSON given a class. That said, I’ve found this library to bit a bit difficult to use for parsing arbitrary JSON structures. It definitely works the best when you use kotlinx.serialization on both the client and the server. I do usually keep
ignoreUnkownKeys: true
and
isLenient: true
whenever I’m not using the same models on the server, because 3rd party servers are the wild west and always end up sending stuff in production that I didn’t expect.
k

kevin.cianfarini

01/31/2023, 10:13 PM
It might not be the “standard” to be so strict, but consider that, unlike most serialization libraries, this one is designed so that you use the same serialization models on both the server and client.
This doesn’t matter when you’re deploying applications to an app store where you have no control over when your clients are actually updated unfortunately.
I’ll happy admit that the blanket “backwards compatible” claim is wrong but I also think the inverse isn’t necessarily true either. There’s no hard spec here for JSON like this is for GRPC
v

Vasileios Chroniadis

01/31/2023, 10:14 PM
Interesting take Casey, ty. I'm thinking Android too here
c

Casey Brooks

01/31/2023, 10:16 PM
That’s a fair point. I can’t tell you how many times my clients have said “can’t you just force update when there’s a change to the API?” No. The answer is always no. A non-compatible change to the API will certainly break Android and iOS apps.
k

kevin.cianfarini

01/31/2023, 10:16 PM
Yup.
It’s unfortunate that the web solved this problem ages ago and greedy companies have regressed us this far.
e

ephemient

01/31/2023, 10:18 PM
the kotlinx.serialization.json default of
ignoreUnknownKeys=false
confuse me, especially since kotlinx.serialization.protobuf has its behavior default to the equivalent of
true
(as is the expectation for that format)
k

kevin.cianfarini

01/31/2023, 10:19 PM
I generally think it’s a bad default, too.
e

Emil Kantis

01/31/2023, 10:21 PM
Even you intend to share models between server/client, you typically can't guarantee simultaneous deployments of client and server..
v

Vasileios Chroniadis

01/31/2023, 10:22 PM
thats true too
k

kevin.cianfarini

01/31/2023, 10:23 PM
The above is very similar to best practice with database migrations. You generally want application code and a migration to be backwards compatible for deploy 1 and then can remove the cruft in deploy 2 after deploy 1 succeeds (assuming a client/server model for an RDBMS).
c

Casey Brooks

01/31/2023, 10:31 PM
So moral of the story when making non-compatible changes to your API: 1) update your client apps with the
car
property in its model 2) Deploy to the app stores 3) Wait 2 weeks for 80% of your apps to be updated 4) Deploy the not-backward-compatible server changes to production 5) 20% of your users start complaining about crashes 6) customer service tells everyone to just update their apps 7) folks still don’t update their apps
k

kevin.cianfarini

01/31/2023, 10:31 PM
step 0: Set
ignoreUnknownKeys
to true and communicate expectations to your backend devs
c

Casey Brooks

01/31/2023, 10:31 PM
Or, just make sure
ignoreUnknownKeys
is true the next time you release, and hopefully don’t have to worry about this as much
v

Vasileios Chroniadis

01/31/2023, 10:31 PM
7 is my favorite
e

ephemient

01/31/2023, 10:32 PM
moral of the story is that you need to set up the apps to allow for future changes, or your changes always go to new endpoints that the older clients aren't using
v

Vasileios Chroniadis

01/31/2023, 10:34 PM
Reading this doc, it reads like: We recognise that "...new properties can be added during the API evolution", but we're gonna crash by default anyways
k

kevin.cianfarini

01/31/2023, 10:34 PM
I think GraphQL does a lot of things poorly, but one thing that’s super nice about it is having a type schema that allows for API evolution and deprecations.
There’s a related API compatibility tool here. I haven’t used it but I became curious. https://github.com/IBM/jsonsubschema
v

Vasileios Chroniadis

01/31/2023, 10:38 PM
I think with GraphQL the clients can select (query) which things from the database they want to consume as JSON. Its just another form of change management, in effect similar to versioning
k

kevin.cianfarini

01/31/2023, 10:41 PM
There’s more to it than that. You can build rich APIs on top of GraphQL. I use it every day.
e

ephemient

01/31/2023, 10:41 PM
there are a couple of pain points with GraphQL - we've run into https://github.com/graphql/graphql-js/issues/1361 more times than I can count, and https://github.com/graphql/graphql-spec/issues/550 makes it harder to deprecate and migrate. but overall it's a reasonably good story
k

kevin.cianfarini

01/31/2023, 10:41 PM
Let me be clear — I hate GraphQL. I just think that having a codified schema with types and deprecations is nice.
I frequently run into the issue that GraphQL cannot express a 64bit integer
e

ephemient

01/31/2023, 10:42 PM
ditto JSON, without extensions that don't work on every platform
c

Casey Brooks

01/31/2023, 10:43 PM
A haven’t used GraphQL in a while, but I also found I didn’t care for it much. Its goal of being language-agnostic actually made it pretty difficult to match its schema semantics into any language except JS (since it’s dynamic). Such is the world we live in. JS is the only supported language for most things, apparently.
k

kevin.cianfarini

01/31/2023, 10:43 PM
Yup. JS and its dumb 53 (I think?) bit numbers integers
Apollo GraphQL on Kotlin has come a long way, but it’s still not perfect.
e

ephemient

01/31/2023, 10:44 PM
IEEE-754 binary64 gives you precise integers up to 2^53, yes
there's integers beyond that, you just lose the ability to represent them all precisely 😛
k

kevin.cianfarini

01/31/2023, 10:45 PM
Which is great for money
e

ephemient

01/31/2023, 10:45 PM
for the record, it's not just a JS issue. Lua does the same thing
k

kevin.cianfarini

01/31/2023, 10:45 PM
Any language which smooshes the semantics of integer types and floating point types into one thing
c

Casey Brooks

01/31/2023, 10:45 PM
You probably shouldn’t be using floating-point numbers for money at all. A 2-integer system is safer and doesn’t lose precision (which is basically what Java’s
BigInteger
does under the hood)
k

kevin.cianfarini

01/31/2023, 10:46 PM
Js just happens to be the one we’re all forced to use
Yup we represent our money as ints
There’s also similar concerns for other units of measurements, like
Watt Hours
here
e

ephemient

01/31/2023, 10:46 PM
at one of my previous companies, all currency was represented in millis as int64
overkill but safe
k

kevin.cianfarini

01/31/2023, 10:47 PM
The GRPC money class is overkill but it’s nice they provide it out of the box
v

Vasileios Chroniadis

01/31/2023, 10:47 PM
I would make an integer joke, but they just don’t have any point
c

Casey Brooks

01/31/2023, 10:47 PM
or alternatively, sometimes “stringly-typed” values are actually the better way to go. Don’t trust the framework to handle numbers correctly, do it all yourself. Painful, but safe
k

kevin.cianfarini

01/31/2023, 10:48 PM
If you can agree on what format to represent your stuff in, maybe. I work for a global org where locale is at the centerpoint of all money representations so I would balk at that idea as dangerous.
e

ephemient

01/31/2023, 10:48 PM
yeah. our current product has integer IDs on the backend but the clients don't need to know that. as far as the clients are concerned, all IDs are simply opaque strings.
k

kevin.cianfarini

01/31/2023, 10:49 PM
e

ephemient

01/31/2023, 10:49 PM
oh nanos is incredibly overkill lol
in any case I would tend to avoid doing anything currency-related on the client anyway. these things change in the real world (e.g. https://en.wikipedia.org/wiki/Croatian_kuna is no more as of a couple weeks ago) and you can update much more consistently on the backend
k

kevin.cianfarini

01/31/2023, 10:53 PM
centerpoint of all money representations
For example, we recently had a prod bug regarding eastern arabic numerals. There’s also some weird stuff with bulgarian where they culturally don’t include a
,
as a grouping separator but only when dealing with money. If it’s normal numbers they do. So strange.
c

Casey Brooks

01/31/2023, 10:55 PM
^ yup, I’ve seen that kind of issue before. In my case, it was a phone set to Turkish locale inserting a
,
instead of
.
for the decimal portion of the USD
e

ephemient

01/31/2023, 10:55 PM
(if that's your issue, you have problems with far more locales than just Turkish)
k

kevin.cianfarini

01/31/2023, 10:56 PM
French intensifies
c

Casey Brooks

01/31/2023, 10:57 PM
Fortunately, my client only has US-based customers dealing in USD, so we could just hardcode the US locale. But don’t get me started on how terrible the API was that we had to deal with money representations like that at all…
k

kevin.cianfarini

01/31/2023, 10:57 PM
Here’s the TL;dr on money formatting. 1. The formatting of the number and the currency symbol relates to the locale a user is trying to view that information in. Eg. Their device locale. 2. The type of currency a customer/user is interacting with has to deal with what country/merchant/entity they’re interacting with.
You shouldn’t hard code us locale because people residing in the US and interacting with US currency can still have their devices set to a different locale (like turkish)
e

ephemient

01/31/2023, 11:02 PM
hard-coding currency per region is also not great: see above where HRK disappeared, or the VEF/VES change a few years ago (not that VES is stable either)
k

kevin.cianfarini

01/31/2023, 11:03 PM
Yup. Currency should generally be delegated from a server
p

pdvrieze

02/01/2023, 2:24 PM
The reason for the default value of
ignoreUnknownKeys
is simple API compatibility. This key was introduced later (per user demand), but allowing the default behavior to change would break the expectations of many existing users. As such it is stuck to
false
until there is some reason to have an API incompatible change.
e

ephemient

02/01/2023, 3:56 PM
wasn't
Json.ignoreUnknownKeys
pre-1.0? there already was a bunch of API-incompatible changes for 1.0
6 Views