CODEWITHSUNDEEP
apache-kafkaavrokafka-consumer-api
Ratha
Ratha

Reputation: 9702

How can i get the Avro schema object from the received message in kafka?

I try to publish/consume my java objects to kafka. I use Avro schema.

My basic program works fine. In my program i use my schema in the producer (for encoding) and consumer (decoding).

If i publish different objects to different topics( eg: 100 topics)at the receiver, i do not know, what type of message i received ?..I would like to get the avro schema from the received byte and would like to use that for decoding.. Is my understand correct? If so, how can i retrieve from the received object?

Upvotes: 0

Views: 5296

Answers (1)

David Griffin
David Griffin

Reputation: 13927

You won't receive the Avro schema in the received bytes -- and you don't really want to. The whole idea with Avro is to separate the schema from the record, so that it is a much more compact format. The way I do it, I have a topic called Schema. The first thing a Kafka consumer process does is to listen to this topic from the beginning and to parse all of the schemas.

Avro schemas are just JSON string objects -- you can just store one schema per record in the Schema topic.

As to figuring out which schema goes with which topic, as I said in a previous answer, you want one schema per topic, no more. So when you parse a message from a specific topic you know exactly what schema applies, because there can be only one.

If you never re-use the schema, you can just name the schema the same as the topic. However, in practice you probably will use the same schema on multiple topics. In which case, you want to have a separate topic that maps Schemas to Topics. You could create an Avro schema like this:

{"name":"SchemaMapping", "type":"record", "fields":[
  {"name":"schemaName", "type":"string"},
  {"name":"topicName", "type":"string"}
]}

You would publish a single record per topic with your Avro-encoded mapping into a special topic -- for example called SchemaMapping -- and after consuming the Schema topic from the beginning, a consumer would listen to SchemaMapping and after that it would know exactly which schema to apply for each topic.

Upvotes: 1

Related Questions