CODEWITHSUNDEEP
avro
Zazaeil
Zazaeil

Reputation: 4109

Avro: is it OK to avoid shared `Schema`s between producers and consumers?

Some well-established messaging system (no matter which one) is used as a "concurrency glue" between numerous small and independent (micro)services.

So there is some distinct place, where messages are produced to and consumed from.

I'm trying out Avro as binary format and its Schemas bother me: I don't like to maintain global store of actual Schemas, I'd like my consumers to assume some properties do exist and if not, simply log such message as "badly formatted" and immideately switch to next one. That implies following: producer(s) will produce using one Schema and consumer will consume using another Schema, which, of course, will share some properties, likely in a different order. Can I achieve that with Avro or am I misusing it?

Upvotes: 0

Views: 49

Answers (1)

Zazaeil
Zazaeil

Reputation: 4109

http://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html:

The only way you can parse this [Avro] binary data is by reading it alongside the schema, and the schema tells you what type to expect next. You need to have the exact same version of the schema as the writer of the data used. If you have the wrong schema, the parser will not be able to make head or tail of the binary data.

So how does Avro support schema evolution? Well, although you need to know the exact schema with which the data was written (the writer’s schema), that doesn’t have to be the same as the schema the consumer is expecting (the reader’s schema). You can actually give two different schemas to the Avro parser, and it uses resolution rules to translate data from the writer schema into the reader schema.

Upvotes: 1

Related Questions