CODEWITHSUNDEEP
apache-kafka
kjoxiphucy
kjoxiphucy

Reputation: 47

Kafka: how does consumer offsets work with dynamically created group ids?

In Kafka, each consumer group is represented by a unique group.id property. Each consumer group manages their own offset (stored in __consumer_offsets topic). What happens to this offset if I always start my consumer service with a dynamically generated group.id value?

Will this new consumer group always read from the beginning of the topic since it has no offset, or will 'auto.offset.reset' take effect?

Upvotes: 2

Views: 1097

Answers (1)

Mickael Maison
Mickael Maison

Reputation: 26885

If you generate a new group.id each time your application starts, the consumer will rely on auto.offset.reset to find its starting position. This is because there won't be any offsets stored as this is a new group.

With auto.offset.reset, you can instruct consumers to either start from the beginning with earliest or end with latest of the logs.

Note that at startup you can also control the position in your application logic and explicitly seek to an arbitrary position based on whatever you want.

A relatively common pattern is to start from a position derived on time, for example seek to 1 hour ago or start of the day. This can be done using offsetsForTimes() and seek().

Upvotes: 2

Related Questions