CODEWITHSUNDEEP
apache-kafka
John Tabernik
John Tabernik

Reputation: 342

Kafka Topics--should I have more or fewer of them?

We are new to Kafka, so I am looking for some high level guidance. We have data for a single entity (we can call it an "Order") that is essentially a number of different entities (we can call one a "Widget" and one a "Gizmo," but there are about 20 different entity types).

Obviously, there is benefit to thinking of Orders as a single topic because all the parts are related to one order. But design wise, does it make more sense for these to be separate topics (Orders, Widgets, Gizmos, etc.)?

There is no direct correlation between the Widgets and Gizmos--the benefit of keeping them together would be things like order of processing, etc. And suggestions or good resources to read would be very helpful. Thanks!

Upvotes: 2

Views: 268

Answers (1)

RANGAPRASAD RAMANUJAM
RANGAPRASAD RAMANUJAM

Reputation: 56

I would recommend initially recording the event as a single atomic message, and not splitting it up into several messages in several topics. It’s best to record events exactly as you receive them, in a form that is as raw as possible. You can always split up the compound event later, using a stream processor—but it’s much harder to reconstruct the original event if you split it up prematurely. Even better, you can give the initial event a unique ID (e.g. a UUID); that way later on when you split the original event into one event for each entity involved, you can carry that ID forward, making the provenance of each event traceable.

Upvotes: 4

Related Questions