CODEWITHSUNDEEP
javaapache-kafkaprotocol-buffersconfluent-schema-registry
ashur
ashur

Reputation: 4307

Do I need schema registry if my team will provide jar library with model?

We are starting new project with Kafka and Protocol Buffers as a data format.

It looks like Serializer/Deserializer provided by Confluent is assuming usage of Schema-Registy, but since my team is going to publish new jar every time a model changes is there any reason to use the registry at all?

I've read multiple articles and sources, but still can't really understand why use schema registry if you have model already. Is it the case when you don't get the jar with classes or any other use-case? Would be great to have some example of this in practice.

Upvotes: 1

Views: 625

Answers (1)

Valath
Valath

Reputation: 920

Even if you have model classes, you may need schema registry. It is more to have a schema evolution, so that no schema change would break existing consumer/producer.

Also a suggestion, irrespective of you use schema registry, I would suggest to use schema(json/avro/protbuf) in the producers and consumers to generate model classes other than adding as a lib. You can get the schema and include a plugin in your build step (maven/gradle) to generate those classes. Always keep a schema as the source of truth for all the discussions even if you are using schema registry or not in Kafka env.

Another big advantage of schema registry is it will reduce the size of the payload over wire and Kafka broker logs. No need to send the entire schema + body, we just send the schema Id + body, it will be decrypted in consumer. This will improve performance.

From an operational point of view, schema registry can act as a single source of truth for schema. Down side is that you have one more component that you need to take care of.

So it depends on your business requirement, you need to make a call based on the plus and minus. But do include only schema in code to generate the model classes on build than committing in source repo/ depending on other team for it.

Please check https://docs.confluent.io/platform/current/schema-registry/index.html https://docs.confluent.io/platform/current/schema-registry/avro.html

Upvotes: 2

Related Questions