Is anyone using avro4k to manage their serialisati...
# announcements
k
Is anyone using avro4k to manage their serialisation to Avro for Kafka? How are you getting it to work with the
KafkaAvroDeserializer
? I'm getting the following error:
Copy code
Unsupported Avro type. Supported types are null, Boolean, Integer, Long, Float, Double, String, byte[] and IndexedRecord
https://github.com/sksamuel/avro4k
r
Are you setting the 5 magic bytes Kafka uses for the schema registry id?
m
@sam ^^^
k
@rocketraman what are the 5 magic bytes? Do I need to build a composite
data class
to wrap them and return my actual
data class
underneath it?
I'm not really sure how this library is supposed to be used if I'm honest. I can se/deserialise raw AVRO binaries, but not sure how it wires up to the Kafka Schema Registry
r
Well, I'm not sure exactly what you're trying to do, but when you read/write Avro to/from Kafka, the built-in serializers store the id of the schema along with the message in the first 5 bytes. If you're trying to replicate what the built-in Kakfa Avro serdes do, then you need to take that into account.
k
At the moment I'm doing
Avro.default.fromRecord(MyDataClass.serializer(), record)
to produce a
GenericRecord
I'm just guessing from your error message you might be reading a message with the magic bytes, which is why you are seeing that error.
s
Yeah Kafka does this trick where it uses the first few bytes to store the “id” of the schema
that id maps to the schema registry
So an Avro message that’s gone through the confluent avro serde is not a pure Avro message anymore it’s “magicbytes” + “avrobytes”
so you can’t just feed it back into an Avro deserializer without removing those bytes
k
Okay, that makes sense
So if I wanted to convert the GenericRecord into something I can use with an avro4k class, I'd need to strip those first 5 bytes?
Or should I first convert it back to
ByteArray
, strip those magic bytes, and then convert it?
s
You would need to strip those bytes yes - or we could add a module to avro4k to do ti
I’m doing something similar in avro4s I think
r
The way I'd approach it is modify Kafka's upstream serializer/deserializer code to use avro4k. All the code is already there to deal with talking to the schema registry, adding the magic bytes at serialization time, and stripping them at deserialization time. And the code is easy to follow.
s
Yeah you could probably just extend the Abstract Deserializer with one that delegates to avro4k
r
Yup
701 Views