CODEWITHSUNDEEP
apache-kafkakafka-consumer-apikafka-producer-api
Mazen Ezzeddine
Mazen Ezzeddine

Reputation: 822

Kafka rebalancing - assignement of Kafka consumers to partitions

When a new Kafka consumer joins/leaves a consumer group, Kafka runtime triggers a rebalancing process so that a new assignment/mapping of partitions to the new set of consumers is performed. I kindly have three questions on the rebalancing process:

(1) Is it possible to plug in somehow a custom rebalancing algorithm, other than the ones already implemented?

(2) Is it possible to pass an external variable to the custom algorithm?

(3) Is it possible to dynamically and selectively launch a rebalancing process, other than the known cases (such as when consumer leaves, join etc..)

Upvotes: 2

Views: 626

Answers (1)

Mickael Maison
Mickael Maison

Reputation: 26865

  1. Yes, you can use partition.assignment.strategy to set a custom assignment algorithm. The custom class needs to implement ConsumerPartitionAssignor.

  2. The assignor can do any necessary logic to retrieve values from the environment or other places.

  3. Since Kafka 2.6, you can use enforceRebalance() to trigger one.

Upvotes: 3

Related Questions