CODEWITHSUNDEEP
scalaapache-sparkapache-kafka
pawan likhi
pawan likhi

Reputation: 41

How to re-consume messages from the beginning?

I am using auto.offset.reset=earliest in my code and used offset commit in kafka with the help of below code.

val offsetRanges=rdd.asInstanceOf[HasOffsetRanges].offsetRanges
inputStream.asInstanceOf[CanCommitOffsets].commitAsync(offsetRanges)

Now when I run my program it does not get new messages as all messages are committed.

I am testing this code in QA so want to reset the offset to beginning but looks like earliest is not working it is not reading new messages and there are no new messages in the topic. I want to read messages from beginning for testing purpose.

Can someone assist if earliest does not fetch messages from beginning if it is committed?

Upvotes: 0

Views: 87

Answers (1)

Roman Kishchenko
Roman Kishchenko

Reputation: 677

The property auto.offset.reset is used only if there's no committed offset for the partition. You can reset the offset for the whole group using kafka-consumer-groups (comes as part of Kafka):

kafka-consumer-groups --bootstrap-server <kafkahost:port> --group <group_id> --topic <topic_name> --reset-offsets --to-earliest --execute

Upvotes: 1

Related Questions