Is anyone using avro4k to manage their serialisation to Avro

Question

Is anyone using avro4k to manage their serialisation to Avro for Kafka How are you getting it to work with the `KafkaAvroDeserializer` I m getting the following error ``` Unsupported Avro type Supported types are null Boolean Integer Long Float Double String byte and IndexedRecord ``` <https github com sksamuel avro4k>

rocketraman · Answer

Are you setting the 5 magic bytes Kafka uses for the schema registry id

Mike Leach · Answer

< sam> ^^^

kushalp · Answer

< rocketraman> what are the 5 magic bytes Do I need to build a composite `data class` to wrap them and return my actual `data class` underneath it

kushalp · Answer

I m not really sure how this library is supposed to be used if I m honest I can se deserialise raw AVRO binaries but not sure how it wires up to the Kafka Schema Registry

rocketraman · Answer

Well I m not sure exactly what you re trying to do but when you read write Avro to from Kafka the built in serializers store the id of the schema along with the message in the first 5 bytes If you re trying to replicate what the built in Kakfa Avro serdes do then you need to take that into account

kushalp · Answer

At the moment I m doing `Avro default fromRecord MyDataClass serializer record ` to produce a `GenericRecord`

rocketraman · Answer

rocketraman · Answer

I m just guessing from your error message you might be reading a message with the magic bytes which is why you are seeing that error

sam · Answer

Yeah Kafka does this trick where it uses the first few bytes to store the id of the schema

sam · Answer

that id maps to the schema registry

sam · Answer

So an Avro message that s gone through the confluent avro serde is not a pure Avro message anymore it s magicbytes + avrobytes

sam · Answer

so you can t just feed it back into an Avro deserializer without removing those bytes

sam · Answer

You can see some of the magic it s doing here <https github com confluentinc schema registry blob ca4f640542316f0bf915ac3ada65ea508147a7e0 avro serializer src main java io confluent kafka serializers AbstractKafkaAvroDeserializer java>

kushalp · Answer

Okay that makes sense

kushalp · Answer

So if I wanted to convert the GenericRecord into something I can use with an avro4k class I d need to strip those first 5 bytes

kushalp · Answer

Or should I first convert it back to `ByteArray` strip those magic bytes and then convert it

sam · Answer

You would need to strip those bytes yes or we could add a module to avro4k to do ti

sam · Answer

I m doing something similar in avro4s I think

rocketraman · Answer

The way I d approach it is modify Kafka s upstream serializer deserializer code to use avro4k All the code is already there to deal with talking to the schema registry adding the magic bytes at serialization time and stripping them at deserialization time And the code is easy to follow

sam · Answer

Yeah you could probably just extend the Abstract Deserializer with one that delegates to avro4k

rocketraman · Answer

Yup