CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
Jan94
Jan94

Reputation: 43

Kafka Consumer Rebalancing and Its Impact

I'm new to Kafka and I'm trying to design a wrapper library in both Java and Go (uses Confluent/Kafka-Go) for Kafka to be used internally. For my use-case, CommitSync is a crucial step and we should do a read only after properly committing the old one. Repeated processing is not a big issue and our client service is idempotent enough. But data loss is a major issue and should not occur.

I will create X number of consumers initially and will keep on polling from them. Hence I would like to know more about the negative scenario's that could happen here, Impact of them and how to properly handle them.

I would like to know more about:

1) Network issue during consumer processing: What happens when network goes of for a brief period and comes back? Does Kafka consumer automatically handle this and becomes alive when network comes back or do we have to reinitialise them? If they come back alive do they resume work from where they left of? Eg: Consumer X read 50 records from Partition Y. Now internally the consumer offset moved to +50. But before committing network issue happens and the comes back alive. Now will the consumer have the metadata about what it read for last poll. Can it go on to commit +50 in offset?

2) Rebalancing in consumer groups. Impact of them on existing consumer process - whether the existing working consumer instance will pause and resume work during a rebalance or do we have to reinitialize them? How long can rebalance occur? If the consumer comes back alive after rebalance, does it have metadata about it last read?

3) What happens when a consumer joins during a rebalancing. Ideally it is again a rebalancing scenario. What will happen now? The existing will be discarded and the new one starts or will wait for the existing rebalance to complete?

Upvotes: 4

Views: 1412

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 192023

What happens when network goes of for a brief period and comes back? Does Kafka consumer automatically handle this and becomes alive when network comes back or do we have to reinitialise them?

The consumer will try to reconnect. If the consumer group coordinator doesn't receive heartbeats or brokers don't respond to brokers, then the group rebalances.

If they come back alive do they resume work from where they left of?

From the last committed offset, yes.

whether the existing working consumer instance will pause and resume work during a rebalance

It will pause and resume. No action needed.

How long can rebalance occur?

Varies on many factors, and can happen indefinitely under certain conditions.

If the consumer comes back alive after rebalance, does it have metadata about it last read?

The last committed offsets are stored on the broker, not by consumers.

The existing will be discarded and the new one starts or will wait for the existing rebalance to complete?

All reblances must complete before any polls continue.

Upvotes: 3

Related Questions