CODEWITHSUNDEEP
c#avro
Loading
Loading

Reputation: 1110

What metadata to add to the front of an avro record to work with kafka?

I have my own avro serializer, but kafka throws this error:

Error deserializing Avro message for id -1","Unknown magic byte!"]},"recordProcessingError":null,"productionError":null} (processing.8236092859265547583.KsqlTopic.source.deserializer:44)

I understand that i need to add the "Magic Byte" and the schema id at the front, but i cant find anything about this "magic byte" or the format of the schema id. I tried the stuff below, but now i get this error:

"message": "Kafka error: Error retrieving Avro schema for id 5202538"

Can you help me here?

This is my (horrible) try:

    var avroRecordWithoutTheMetadata = data.ToArray().ToList();
    var schemaIdBytes = BitConverter.GetBytes(Program.schemaID).ToList();
    var magicByte = new byte();

    var correctAvroRecord = new List<byte>() {magicByte};
    correctAvroRecord.AddRange(schemaIdBytes);
    correctAvroRecord.AddRange(avroRecordWithoutTheMetadata);

    var result = correctAvroRecord.ToArray();

Upvotes: 0

Views: 563

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191854

i cant find anything about this "magic byte" or the format of the schema id

The magic byte is just 0x0 and the schema id is the next 4 bytes as an integer.

https://docs.confluent.io/current/schema-registry/docs/serializer-formatter.html#wire-format

Alternatively, just use the existing serializers provided by Confluent

Upvotes: 1

Related Questions