CODEWITHSUNDEEP
javaspring-bootapache-kafka
Tariq Abbas
Tariq Abbas

Reputation: 171

Start consuming only latest messages from Kafka Topic by Ignoring all existing messages

how can i consume only latest messages from kafka topic by ignoring all existing messages in the topic. I have two consumers of the same topic and when i start consuming messages from the topic it fetches the oldest messages. I need to consume messages after my consumer started. I tried this configuration in my Consumer configuration but this is not working.

config.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");
config.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");

Upvotes: 5

Views: 9784

Answers (3)

jrg
jrg

Reputation: 303

You need to ensure your Consumer (or any earlier instance) did not commit any offset for your consumer-group, if you want to ignore all existing messages.

Configuring AUTO_OFFSET_RESET_CONFIG = "latest" works as long as there are no committed offsets. If there are, they will always override:

What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted):

  • earliest: automatically reset the offset to the earliest offset
  • latest: automatically reset the offset to the latest offset
  • none: throw exception to the consumer if no previous offset is found for the consumer's group
  • anything else: throw exception to the consumer.

If you are not able to rely upon there not being any committed offsets then use seekToEnd() on your assigned partitions before you start polling.

Upvotes: 3

Tariq Abbas
Tariq Abbas

Reputation: 171

auto.offset.reset

after setting above parameter when you don't change your groud.id i will again start reading from message 0. if you want to read current messages from topic , first set:

  config.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");

and then change you group.id and redeploy your consumer. it will start consuming latest messages.

Upvotes: 3

riorio
riorio

Reputation: 6816

Did you try

auto.offset.reset

https://www.oreilly.com/library/view/kafka-the-definitive/9781491936153/ch04.html

Upvotes: -1

Related Questions