CODEWITHSUNDEEP
apache-kafka
Glide
Glide

Reputation: 21275

Kafka how to preserve order of events when partitions increase

I want to create N partitions of user events by modding user_id by N so events can be processed in the order by the consumers in the order they were sent.

If I ever decide N is not enough to handle the load and want to increase the number of partitions and consumers respectively, what do I have to do to preserve the events order when consuming user events?

Upvotes: 0

Views: 1675

Answers (1)

Aurand
Aurand

Reputation: 5547

Well, you could create a new topic with an increased partition count and then copy all your events over into the new topic. That way you maintain ordering (with respect to a given user_id, there would be no promises about ordering across different user_id in your original scheme anyway).

Of course, that's likely to require downtime. The naive solution of just increasing the partition count obviously won't work since it will change your hashing calculation and result in the events for a given user_id being split across multiple partitions (and thus losing ordering). The difficulty of increasing partition count is one of the reasons you want to think hard about partition count when initially creating a topic.

Upvotes: 1

Related Questions