CODEWITHSUNDEEP
apache-kafkaspring-kafka
Aja Hebbe
Aja Hebbe

Reputation: 13

How to get only latest message when re-connecting to an existing consumer group in Kafka

In my case, it is a valid possibility that a consumer is offline for a longer period. During that offline period, events are still published to the topic.

When the consumer comes back online, it will re-use its existing consumer group, which has been lagging. Is it possible to skip forward to the latest message only? That is, ignore all earlier messages. In other words, I want to alter the offset to the latest message prior to consuming.

There is the spring.kafka.consumer.auto-offset property, but as far as I understand, this is only applicable for new consumer groups. Here, I am re-using an existing consumer group when the consumer comes back online. That said, if there is a possibility to automatically prune a consumer group when its consumer goes offline, this property could work, but I am not sure if such functionality exists?

I am working with the Spring Boot Kafka integration.

Upvotes: 1

Views: 1305

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191743

You can use consumer seek method after you calculate the last offset then subtract one from that, commit, and start polling.

Otherwise, simply don't use the same group. Generate a unique one and/or disable auto commits, then you're guaranteed to always use the auto.offset.reset config and lag is meaningless across app restarts

Upvotes: 1

Related Questions