CODEWITHSUNDEEP
javaapache-kafka
NSA
NSA

Reputation: 6057

When using the Kafka high level consumer is there a way to be notified of a rebalancing?

When using the java Kafka(0.8) high level consumer is there a way to determine that a rebalance has, or needs to occur?

Thank you!

Upvotes: 0

Views: 249

Answers (2)

Denis Makarenko
Denis Makarenko

Reputation: 2938

As Cesar mentioned above, there is no clean way to get notified about rebalancing. However note that rebalancing happens when a reader or broker connects or disconnects and also when the number of partitions for a topic changes. So you can monitor Zookeeper nodes where brokers and readers are registered. Note that it may be brittle though as the Kafka related ZK node layout may change in future Kafka versions.

See Broker Node Registry, Consumer Id Registry and Partition Owner registry sections in Kafka docs

Also, it is not immediately obvious but rebalancing may fail and high level consumer may throw ConsumerRebalanceFailedException. So it is something you need to take into consideration as well. See Kafka rebalancing for a high level overview of rebalancing process.

Upvotes: 1

Cesar Villasana
Cesar Villasana

Reputation: 687

Kafka does this automatically, There is no way to determine this except for monitoring the logs.. What you can do is something like

tail -f <myKafkaLogFiles> | grep rebalance

I don't remember if the logs print the word "rebalance" or "rebalancing", maybe try with both words and check. The command above will only print logs on screen that match the word passed as an argument to the grep command..

Hope it helps.

Upvotes: 0

Related Questions