Can anyone here tell me how significant the binary compatibility issues are for `NonEmptyList` (usin...

# arrow

[Stephen Morse]

Can anyone here tell me how significant the binary compatibility issues are for NonEmptyList (using NonEmptyListSerializer) between Arrow v1 and v2? My company (Cash App/Block) has some data serialized with the NonEmptyListSerializer of Arrow v1, but we need to upgrade to Arrow v2. Is it possible to define a serializer that can deserialize both the v1 and v2 binary formats? 🤔

[Stephen Morse]

From the Arrow 2.0 Release blog:

we had to break binary compatibility in several places to implement improvements, such as in NonEmptyList

[Youssef Shoaib [MOD]]

I don't have the source in front of me right now, but I'd guess it's likely fine? If not though, you could write a program to deserialize it, convert it to List , serialize it again, deserialize in 2.0, and finally convert it to NonEmptyList and serialize

[phldavies]

we had to break binary compatibility in several places to implement improvements, such as in NonEmptyList

this is referencing the binary compatibility of the arrow-core library itself, not the kotlinx-serialization serialized content (assuming that's what you mean by serialized)

The incompatibility would arise should you try and execute a binary/jar compiled against arrow v1 with arrow v2 on the classpath

[phldavies]

FYI NonEmptyListSerializer is a simple wrapper around a delegated ListSerializer with the additional conversion to NonEmptyList using .toNonEmptyListOrNull() ?: throw SerializationException("expected non-empty list") when deserializing

[Ulrich Schuster]

We did run into some serious issues, especially in combination with Spring "magic" – e.g., when using a NonEmptyList for API parameters or similar data structures where Spring attempts to create proxies.

[phldavies]

Yes, this is likely due to the change to using a value class, and a kotlin compiler bug that leaves type-args in when it shouldn't (which impacts proxies etc)

[Alejandro Serrano.Mena]

btw, the bug in the compiler has been fixed, but will only appear in 2.2.20 https://youtrack.jetbrains.com/issue/KT-52706/Bad-signature-for-generic-value-classes-with-substituted-type-parameter

[Alejandro Serrano.Mena]

in this case I think that:
• binary compatibility is definitely not there, since the underlying format of NonEmptyList has changed
• on the other hand, serializing in v1 and deserializing in v2 should work fine (or even the other way around), since both turn into the same thing