Hi everyone :wave: new to kotlin, coming from scal...

# arrow

[João Horta Alves]

Hi everyone 👋 new to kotlin, coming from scala. I'm exploring Arrow + Ktor. Can you clarify something for me, I might be missing something but it looks like there's no clean way to do call.receive<NonEmptyList<...>> in ktor with arrow-core-serialization.
Meaning that the mechanism for parsing JSON relies on the @Serializable annotation, and since NonEmptyList is not annotated I cannot just parse it directly, even using @file:UseSerializers(NonEmptyListSerializer::class) on the file?
Also posted to stack overflow.

[João Horta Alves]

I've also tried using this approach, but it doesn't work with ktor call.receive though it works with plain Json.decodeFromString

[simon.vergauwen]

Aha okay, that's something else ☺️ That is KotlinX Serialization specific problem..

KotlinX uses reflection to find serializers, but it doesn't do classpath scanning so it cannot automatically detect NonEmptyListSerializer. @file only works when you add it in a file that defines the data class like you said.

So for KotlinX to find the polymorphic serializer, you need to use KotlinX Serializers Module. https://github.com/Kotlin/kotlinx.serialization/blob/master/docs/polymorphism.md#merging-library-serializers-modules

So in Ktor:

install(ContentNegotation) {
  json {
     serializersModule = SerializersModule {
     }
  }
}

[João Horta Alves]

Ah so maybe there's a clean way! Thanks for the pointer, I'm reading the docs but not clear yet to me how to apply to this case...

[João Horta Alves]

That shows how to register subclasses which are already annotated with Serializable, not sure how that helps me register NonEmptyListSerializer

[simon.vergauwen]

I've needed this in the past for List<MyClass> as well, since List is not annotated with @Serializable and/or because A cannot be verified to be @Serialiazer without the SerializersModule. Sorry, I am not an expert on this area but afaik it's related to the runtime reflection lookup

[João Horta Alves]

ok, I think I might have figured it out:

install(ContentNegotiation) {
        json(Json {
            serializersModule =
                SerializersModule { contextual(NonEmptyList::class) { args -> NonEmptyListSerializer(args[0]) } }
        })
    }

https://github.com/Kotlin/kotlinx.serialization/blob/master/docs/serializers.md#contextual-serialization-and-generic-classes

[João Horta Alves]

Thanks for the help!

[João Horta Alves]

though it would be nice to have this documented somewhere, it's very non-obvious 😕

[João Horta Alves]

Anyway I'll leave it as an answer in stack overflow

[simon.vergauwen]

We should add this to the website, it's indeed not clear. We didn't document it because it's KotlinX behavior, but I agree it's so cumbersome it's worth adding to our docs.

If you're interested in adding a small section as a contribution, https://github.com/arrow-kt/arrow-website/blob/main/content/docs/learn/quickstart/serialization.md

[João Horta Alves]

I'll look into it when I find some time 🙂

[simon.vergauwen]

Awesome! If you have any other feedback coming from Scala, I'd love to hear it btw! 😉

[simon.vergauwen]

@Garth Gilmour wanted to let you know this wasn't covered in your blogpost either. We link it from the website 😉
https://garthgilmour.medium.com/marshalling-arrow-types-in-ktor-bc471aa3650

[João Horta Alves]

I'm guessing that this approach of registering the serializers could be a (more powerful) alternative to using @file:UseSerializers, at least in Ktor

[simon.vergauwen]

I'm not entirely sure there is no difference between the two 🤔

IIRC @file:UserSerializers is used inside the generated code, but serializers module affects runtime lookup.

[simon.vergauwen]

And/Or, @file:UserSerializers is required for example for val instant: org.jebtrains.kotlinx.datetime.Instant.

[João Horta Alves]

ah ok, it is different, if I add a NonEmptyList to a Serializable data class, compiler complains Serializer has not been found for type 'NonEmptyList<String>'. because the SerializersModule mechanism must be only runtime

[simon.vergauwen]

Yes, so there you still need @file:UseSerializers(NonEmptyListSerializer::class) which the compiler will pick-up while generating the serializer for the data class

[João Horta Alves]

I'm definitely noticing more runtime stuff than in FP Scala, but I'm happy at least there was a clean solution, even if it's runtime 😄

[Garth Gilmour]

> wanted to let you know this wasn't covered in your blogpost either.
Thanks @simon.vergauwen! When i get the time will do an update.

[phldavies]

Is it worth adding a SerializersModule to arrow-core-serialization similar to

val ArrowSerializersModule = SerializersModule {
    contextual(Either::class) { (a, b) -> EitherSerializer(a, b) }
    contextual(Ior::class) { (a, b) -> IorSerializer(a, b) }
    contextual(NonEmptyList::class) { (t) -> NonEmptyListSerializer(t) }
    contextual(NonEmptySet::class) { (t) -> NonEmptySetSerializer(t) }
    // contextual(Option::class) { (t) -> OptionSerializer(t) } // mismatch on T vs T: Any
    @Suppress("DEPRECATION")
    contextual(Validated::class) { (a, b) -> ValidatedSerializer(a, b) }
}

allowing consumers to just use

install(ContentNegotiation) {
    json(Json {
        serializersModule = SerializersModule {
            include(ArrowSerializersModule)
        }
    })
}

?

[simon.vergauwen]

That is a great idea @phldavies!

[phldavies]

Happy to throw a PR up, but I'm not sure how best to handle the Option<T> vs OptionSerializer<T: Any> discrepancy

[simon.vergauwen]

Uh, I am not aware of that 😅 A PR would be awesome, ignore Option for now in the PR and we'll while go.

[phldavies]

So the OptionSerializer being declared as <T: Any> is ignored by kotlinx-serialization.

@Serializable
data class MyData(val a: @Serializable(with=OptionSerializer::class) Option<String?>)

class Example {
  @Test fun example() {
    val json = Json.encodeToJsonElement(MyData(Some(null)))
    println(json)
    Json.decodeFromJsonElement<MyData>(json) shouldBe MyData(Some(null))
  }
}

This compiles fine, despite the discrepancy but the current implementation of OptionSerializer doesn't pass this test (I kinda don't expect it to either as JSON has no nested-nullability)

It seems the OptionSerializer only has <T: Any> to allow it to access the KSerializer<T: Any>.nullable property, but we can work around this with the following:

@Suppress("UNCHECKED_CAST")
@OptIn(ExperimentalSerializationApi::class)
private val <T> KSerializer<T>.nullable get() =
  if (descriptor.isNullable) (this as KSerializer<T?>)
  else (this as KSerializer<T & Any>).nullable

Which means we can lift the restriction on OptionSerializer<T> and use it in contextual, given as the restriction is ignored anyway. The only real impact of the restriction is not being able to use OptionSerializer(String.serializer().nullable)

[phldavies]

PR is up https://github.com/arrow-kt/arrow/pull/3413 - includes Option but can walk that back out if needed