CODEWITHSUNDEEP
apache-kafka
Mandroid
Mandroid

Reputation: 7504

Kafka: change in consumers number in a group

I understand that Kafka semantics is that a consumer group must read a record only once. To achieve this, Kafka consumers maintain an offset, which is then conveyed to brokers with read requests so that brokers can send data accordingly to ensure that already read data is not resend(). But how does broker and consumers react when their is a change in consumer group, like addition of a new consumer or an existing consumer going down?

There are few things which needs to be considered here.

Upvotes: 1

Views: 49

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191738

If consumers join/leave a group, there's a consumer group rebalance. All consumers in the group will temporarily be suspended, then new partitions will be assigned to consume from.

If those consumers were processing, then there's a good chance that they'll re-consume the same data.

If you use transactions, the chance that happens could be a reduced as records will be consumed "exactly once". But this doesn't necessarily mean "successfully processed and offset committed" exactly once.

Upvotes: 1

Related Questions