CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
Karthikeyan Rasipalay Durairaj
Karthikeyan Rasipalay Durairaj

Reputation: 2334

Difference between Consumer level offset and Consumer group offset in Kafka

Trying to understand between Offset in Consumer and offset of Consumer group .

Below stack overflow link provides excellent understanding on Consumer group offset management .
What determines Kafka consumer offset? Now question here ,

Scenarios:

We have consumer (c1) in a consumer group group1 .

  1. Is it offset value will be stored in consumer (c1) and group (group1) both level ? Or if consumer will be belong to any consumer group , offset will be stored in Only Consumer group level ?
  2. If offset value will be stored in both level , will it be consumer level offset value will be override consumer group level offset value .

Example

Consumer level offset value - 5 Consumer group level offset value - 8 When system will be restarted from 8 ?

Reverse:

Consumer level offset value - 8 Consumer group level offset value - 5 When system will be restarted from 5 ?

Upvotes: 2

Views: 1176

Answers (1)

Ryuzaki L
Ryuzaki L

Reputation: 40078

It depends on Kafka consumer group management, if the consumer group management is enabled with group.id the offset will be stored in consumer group level, so that in case of re balance a new consumer can read from offset.

And if consumer group management is disabled offset is stored under consume, be default group.id is null if user don't provide it, So there is no chance of offset storing on both the levels

Offset expiration semantics has slightly changed in this version. According to the new semantics, offsets of partitions in a group will not be removed while the group is subscribed to the corresponding topic and is still active (has active consumers). If group becomes empty all its offsets will be removed after default offset retention period (or the one set by broker) has passed (unless the group becomes active again). Offsets associated with standalone (simple) consumers, that do not use Kafka group management, will be removed after default offset retention period (or the one set by broker) has passed since their last commit.

group.id:

A unique string that identifies the consumer group this consumer belongs to. This property is required if the consumer uses either the group management functionality by using subscribe(topic) or the Kafka-based offset management strategy.

Type: stringDefault: nullValid Values:Importance: high

Upvotes: 1

Related Questions