CODEWITHSUNDEEP
apache-kafkaspring-kafka
amandeep1991
amandeep1991

Reputation: 1394

Maintaining ordering of msgs while adding partitions in running cluster

Can we maintain ordering of msgs while adding partitions in running cluster?

May be earlier there were 2 partitions and 4 types of msgs as per partition key and we got ideal key hashing algo so 2 type of msgs (partition keys) were assigned to each of partition.

Adding 2 new partitions (while cluster is still up & running) will start adding new msgs to old & new partitions (as per new key hashing algo), but there is a msg of type#4 in partition#2 at offset 2000 and current offset of consumer for partition#2 is just 10 and new type#4 msg is added in partition#4 which will be consumed first, right?

But I want to maintain the order, is it possible?

Does kafka provide some mechanism for this sort of ordering or I have to do custom handling at consumer side if I want to have this feature/support of maintaining the order while adding new partitions to a running cluster?

Upvotes: 1

Views: 34

Answers (1)

Gary Russell
Gary Russell

Reputation: 174769

You can use a custom Partitioner to determine the target partition.

https://kafka.apache.org/documentation/#producerconfigs_partitioner.class

You could delegate to a default partitioner initially, but cache the destination partition for a particular key.

You would need to persist the cache someplace (perhaps in a compacted topic) if you want it to live over restarts or if you have multiple producer instances.

Upvotes: 2

Related Questions