I'm using protobufs for the first time. A project ...

# serialization

[spierce7]

I'm using protobufs for the first time. A project is sending a oneof protobufs to me that I need to deserialize:

message DeviceConnected {
  string serialNumber = 1;
  int32 vendorId = 2;
  int32 productId = 3;
}

message DataFromDevice {
  string serialNumber = 1;
  bytes data = 2;
}

message DataToDeviceResult {
  string correlationId = 1;
  bool success = 2;
}

message ServerMessage {
  oneof message {
    DeviceConnected deviceConnected = 1;
    DataFromDevice fromDevice = 2;
    DataToDeviceResult toDeviceResult = 3;
  }
}

How would I represent these in Kotlinx.Serialization? The attempts I've tried have resulted in serialization errors, and I really am not sure how to represent this polymorphic proto.

[Dariusz Kuc]

looks like it is doable -> https://github.com/Kotlin/kotlinx.serialization/issues/67

[spierce7]

I tried that approach, but not getting success:

import kotlinx.serialization.Polymorphic
import kotlinx.serialization.Serializable
import kotlinx.serialization.decodeFromByteArray
import kotlinx.serialization.encodeToByteArray
import kotlinx.serialization.modules.SerializersModule
import kotlinx.serialization.modules.polymorphic
import kotlinx.serialization.modules.subclass
import kotlinx.serialization.protobuf.ProtoBuf
import org.junit.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue

class ExampleTest {
    @Test
    fun testSealedInterface() { // Put this function in a class in test sources.
        val module = SerializersModule {
            polymorphic(ServerMessage::class) {
                subclass(ServerMessage.DeviceConnected::class)
                subclass(ServerMessage.DataFromDevice::class)
                subclass(ServerMessage.DataToDeviceResult::class)
            }
        }
        val protobuf = ProtoBuf { serializersModule = module }

        ServerMessage.DeviceConnected("test", 1, 2).let { initial ->
            val byteArray = protobuf.encodeToByteArray(initial)
            val deserialized = protobuf.decodeFromByteArray<ServerMessage>(byteArray)

            assertTrue(deserialized is ServerMessage.DeviceConnected)
            assertEquals("test", deserialized.serialNumber)
        }
    }
}
@Serializable
@Polymorphic
sealed interface ServerMessage {
    @Serializable
    data class DeviceConnected(
        val serialNumber: String,
        val vendorId: Int,
        val productId: Int
    ) : ServerMessage

    @Serializable
    data class DataFromDevice(
        val serialNumber: String,
        val data: ByteArray
    ) : ServerMessage

    @Serializable
    data class DataToDeviceResult(
        val correlationId: String,
        val success: Boolean
    ) : ServerMessage
}

Yields:

kotlinx.serialization.SerializationException: Class 'test' is not registered for polymorphic serialization in the scope of 'ServerMessage'.
To be registered automatically, class 'test' has to be '@Serializable', and the base class 'ServerMessage' has to be sealed and '@Serializable'.
Alternatively, register the serializer for 'test' explicitly in a corresponding SerializersModule.
	at kotlinx.serialization.internal.AbstractPolymorphicSerializerKt.throwSubtypeNotRegistered(AbstractPolymorphicSerializer.kt:102)
	at kotlinx.serialization.PolymorphicSerializerKt.findPolymorphicSerializer(PolymorphicSerializer.kt:102)
	at kotlinx.serialization.internal.AbstractPolymorphicSerializer.deserialize(AbstractPolymorphicSerializer.kt:56)
	at kotlinx.serialization.protobuf.internal.ProtobufDecoder.decodeSerializableValue(ProtobufDecoding.kt:196)
	at kotlinx.serialization.protobuf.internal.ProtobufDecoder.decodeSerializableValue(ProtobufDecoding.kt:186)
	at kotlinx.serialization.protobuf.ProtoBuf.decodeFromByteArray(ProtoBuf.kt:137)
	at ExampleTest.testSealedInterface(ExampleTest.kt:74)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40)
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:60)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at jdk.proxy1/jdk.proxy1.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
	at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
	at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)

I'm not sure what's going on. There is no class test

[glureau]

Polymorphic requires a class discriminator. If you use the protobuf schema generator (from kotlinx serialization) with a sealed class, you'll get a message with 2 fields (string + bytes), the string will be the discriminator, and the bytes the data to decode with the right serializer (given the string). So from my understanding, this message is not compatible with a oneOf.

[glureau]

Note that in the example you've tried, there is this line :

TestInt(id = "abc", 7).let<TestOneOf, _> { initial ->

and this is followed by protobuf.encodeToByteArray(initial).
But this method I believe is a reified one using the type of the parameter, so in the github issue it's equivalent to protobuf.encodeToByteArray<TestOneOf>(initial) , aka the sealed class, and not one of its subclass. You may want to try to specify the generics in your code.

[glureau]

Eventually the example from Louis is doing encode/decode on the same protobuf serializer (so I bet using an "internal" field / class discriminator like he mention at the end of the thread). But in your original message you mention "a project is sending" so my understanding is that you don't have control on this part, and if it's a oneOf, so there will be no discriminator for that.
In this case, it may be better to use bezmax approach, get all the data in a deserialization class, then select the subclass you want manually.

[Chad Gregory]

Could you not include a .proto file with that protobuf definition and have the gradle plugin generate it for you? I have done this for other predefined specs like Sparkplug B

[glureau]

@Chad Gregory Are you mixing generated code from protoc with Kotlinx Serialization for protobuf? Or are you suggesting to only use protoc (instead of KotlinxSerialization)?

[Chad Gregory]

There is a plugin that generates kotlin code from .proto files.

In your build.gradle

plugins {
    kotlin("plugin.serialization") version "1.8.22"
    id("com.google.protobuf") version "0.9.4"
}

dependencies {
    implementation("com.google.protobuf:protobuf-kotlin:3.24.0")
}

protobuf {
    protoc {
        // The artifact spec for the Protobuf Compiler
        artifact = "com.google.protobuf:protoc:3.24.0"
    }

    generateProtoTasks {
        all().forEach { task ->
            task.plugins{
                create("kotlin")
                create("java")
            }
        }
    }
}

Place your .proto file in app/source/main/proto

When you build your project it will generate the code to build/inject data based on that .proto file

[spierce7]

@Chad Gregory Will this code work multiplatform?

[glureau]

Nop it won't, the kotlin export only generates a thin wrapper on top of the java one, so only JVM. You could also generate the objectiveC code and then adds expect/actual or interfaces to make everything works together, but it's painful I think.

[Chad Gregory]

Would be nice to have a kotlin only solution for KMM.