CODEWITHSUNDEEP
apache-kafka
David Faizulaev
David Faizulaev

Reputation: 5721

Kafka consumer - how does rebalance work if one consumer fails

I'm using AWS Kafka MSK and I have a topic with 2 partitions. I also have a 2 consumers which are part of the same consumer group.

I'm wondering that will happen in the following case:

Consumer A - took messages 1 - 100
Consumer B - took messages 101 - 200

Consumer A failed
Consumer B succeeded

What happens to the messages 1 - 100?

Will the auto Kafka rebalance set consumer B to read messages 1 - 100? or the new consumer that will startup instead of Consumer A will read the messages?

Thanks in advance.

Upvotes: 1

Views: 1177

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191671

Offset ranges are for partitions, not topics.

This scenario is not possible for a fresh consumer application unless one of the following is true

  • Offsets 0-100 of the partition assigned to consumer B have been removed due to retention
  • Your code calls seek method to skip those offsets

On the other hand, if the consumer group already existed and consumed none of the records of partition assigned to consumer A (say, it had failed before), and did commit offset 100 of the other partition. In this case, perhaps the same thing would happen; the consumer group might fail reading offset 0 of the "first" partition.

When any consumer instance fails, the group will rebalance. Depending on how you handle errors/failures, the previously healthy instance may then be assigned both partitions, and then fail consuming the "first" partition again (since it'll be the same code that died previously). Or, writing code differently, you'll ignore consumer exceptions and optionally mark bad offsets in a dead-letter queue. When logged or ignored, you'd commit offsets for the original consumer and skip those records.

Upvotes: 3

Related Questions