CODEWITHSUNDEEP
apache-kafkalibrdkafka
Youli Luo
Youli Luo

Reputation: 177

Does Kafka Consumer reconnect and resubscribe to topics after cluster goes down and comes back up

kafka consumer using librdkafka (high level consumer) connected to kafka cluster and subscribed to 10 topics and consuming data. There was assign partitions event.

There was network issue due to which cluster was not reachable. Lost connection with group co-ordinator and heartbeats got stuck. There was revoked partitions event where the code calls unassign on consumer.

When cluster came back up, the consumer was not consuming any data although its in the while true loop calling consume with timeout of 1 sec.

Does consumer needs to resubscribe to topics again once it is connected to cluster? What is a reliable way to detect the consumer is connected to cluster in code?

Upvotes: 0

Views: 2004

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191711

Does consumer needs to resubscribe to topics again once it is connected to cluster?

Yes. New group members will cause a rebalance amongst existing members, and they need to resubscribe

What is a reliable way to detect the consumer is connected to cluster in code?

You could describe the consumer group and see if there are active clients for the consumer group you are interested in

Upvotes: 2

Related Questions