CODEWITHSUNDEEP
apache-kafka
Ronak Kalyani
Ronak Kalyani

Reputation: 113

kafka Partition Rebalance

lets assume, A consumer group having 5 consumers, subscribes to some topic which has 10 partitions. Now if 10 more partitions are added to the same topic. It triggers re-balance. Why kafka reassigns previously assigned partitions, why it can't just assign newly created partitions among consumer group.

Upvotes: 3

Views: 1092

Answers (2)

Trent Bartlem
Trent Bartlem

Reputation: 2253

Note that there are better assignors now, e.g. the CooperativeStickyAssignor.

Details: https://www.confluent.io/blog/cooperative-rebalancing-in-kafka-streams-consumer-ksqldb/

Upvotes: 0

Mickael Maison
Mickael Maison

Reputation: 26920

Due to the way Kafka partition assignors currently work, consumer rebalances are always a "stop the world" operation during which all partitions are first released from consumers before being reassigned.

That said, there is the StickyAssignor which attempts to preserve the previous assignment whereas the default assignor RangeAssignor gives no guarantees.

To use StickyAssignor, set partition.assignment.strategy to org.apache.kafka.clients.consumer.StickyAssignor in your Consumer configuration.

Upvotes: 3

Related Questions