CODEWITHSUNDEEP
goapache-kafkakafka-consumer-apisarama
Sai Satwik Kuppili
Sai Satwik Kuppili

Reputation: 175

How to change offset of a topic during runtime?

I have a Producer for kafka topic which keeps on pushing some messages to kafka topic. And also I have another service reading these messages from topic.

I have an business use-case, where sometimes consumer need to ignore all the messages which are already there in queue and start processing only new upcoming messages. Can this be archived without stopping and restarting the kafka server.

I am working on GO. So if kafka supports such requirement, is there any way I can change configuration of consumer to start consuming from latest message using sarama GO client.

Thank you in advance.

Upvotes: 0

Views: 1023

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191874

You could use a random UUID for consumer group id, and/or disable auto commits, then you can start at the latest offset with

config := sarama.NewConfig()
config.Consumer.Offsets.Initial = sarama.OffsetOldest

(copied from Sarama example code)

Otherwise, Kafka consumer API should have a seekToEnd function, but it seems to be exposed in Sarama as getting high watermarks from consumer for every partition, then calling ResetOffets on a ConsumerGroup instance. Note: the group should be paused before doing that.

Upvotes: 1

Related Questions