CODEWITHSUNDEEP
kubernetesapache-kafkakafka-consumer-api
Mazen Ezzeddine
Mazen Ezzeddine

Reputation: 822

Create a new consumer for a consumer group from within the assign method of the rebalancing interface (PartitionAssignor )

I am running a Kafka cluster on Kubernetes. I am implementing a custom PartitionAssignor to personalize the way topic partitions are assigned to existing consumers in the consumer group. To this end, I am overriding the method Map<String, Assignment> assign( Cluster metadata Map<String, Subscription> subscriptions)

If inside the assign method I dynamically created a new consumer through the Kubernetes client APIs, how would the rebalancing protocol behave in such case. Precisely, when the newly created consumer send a joinGroup request to the group coordinator(while the rebalancing process is still in progress), would the current in progress rebalancing completes, and then a new rebalance process is triggered to accommodate for the newly created consumer?

Upvotes: 0

Views: 160

Answers (1)

Mazen Ezzeddine
Mazen Ezzeddine

Reputation: 822

For those that might be interested in the answer, I posted the question in Kafka mailing list, and below is the answer from Confluent People:

Answer :

If you create a new consumer inside the assign method (and assuming you actually start polling with it so that it sends the JoinGroup request), then yes, it would need a new rebalance to accommodate this consumer. The group coordinator will inform all the existing members to rejoin the group so that the rebalance can proceed with the latest up-to-date view of the current group.

Upvotes: 0

Related Questions