Is anyone using avro4k to manage their serialisation to Avro for Kafka? How are you getting it to wo...

# announcements

[kushalp]

Is anyone using avro4k to manage their serialisation to Avro for Kafka? How are you getting it to work with the KafkaAvroDeserializer ? I'm getting the following error:

Unsupported Avro type. Supported types are null, Boolean, Integer, Long, Float, Double, String, byte[] and IndexedRecord

https://github.com/sksamuel/avro4k

[rocketraman]

Are you setting the 5 magic bytes Kafka uses for the schema registry id?

[Mike Leach]

@sam ^^^

[kushalp]

@rocketraman what are the 5 magic bytes? Do I need to build a composite data class to wrap them and return my actual data class underneath it?

[kushalp]

I'm not really sure how this library is supposed to be used if I'm honest. I can se/deserialise raw AVRO binaries, but not sure how it wires up to the Kafka Schema Registry

[rocketraman]

Well, I'm not sure exactly what you're trying to do, but when you read/write Avro to/from Kafka, the built-in serializers store the id of the schema along with the message in the first 5 bytes. If you're trying to replicate what the built-in Kakfa Avro serdes do, then you need to take that into account.

[kushalp]

At the moment I'm doing Avro.default.fromRecord(MyDataClass.serializer(), record) to produce a GenericRecord

[rocketraman]

https://docs.confluent.io/current/schema-registry/index.html

[rocketraman]

I'm just guessing from your error message you might be reading a message with the magic bytes, which is why you are seeing that error.

[sam]

Yeah Kafka does this trick where it uses the first few bytes to store the “id” of the schema

[sam]

that id maps to the schema registry

[sam]

So an Avro message that’s gone through the confluent avro serde is not a pure Avro message anymore it’s “magicbytes” + “avrobytes”

[sam]

so you can’t just feed it back into an Avro deserializer without removing those bytes

[sam]

You can see some of the magic it’s doing here: https://github.com/confluentinc/schema-registry/blob/ca4f640542316f0bf915ac3ada65ea508147a7e0/avro-serializer/src/main/java/io/confluent/kafka/serializers/AbstractKafkaAvroDeserializer.java

[kushalp]

Okay, that makes sense

[kushalp]

So if I wanted to convert the GenericRecord into something I can use with an avro4k class, I'd need to strip those first 5 bytes?

[kushalp]

Or should I first convert it back to ByteArray, strip those magic bytes, and then convert it?

[sam]

You would need to strip those bytes yes - or we could add a module to avro4k to do ti

[sam]

I’m doing something similar in avro4s I think

[rocketraman]

The way I'd approach it is modify Kafka's upstream serializer/deserializer code to use avro4k. All the code is already there to deal with talking to the schema registry, adding the magic bytes at serialization time, and stripping them at deserialization time. And the code is easy to follow.

[sam]

Yeah you could probably just extend the Abstract Deserializer with one that delegates to avro4k

[rocketraman]

Yup