CODEWITHSUNDEEP
apache-kafka
Adib Rastegarnia
Adib Rastegarnia

Reputation: 574

Kafka producer buffering

Suppose there is a producer which is running and I run a consumer a few minutes later. I noticed that the consumer will consume old messages that has been produced by the producer but I don't want that happens. How can I do that? Is there any config parameters in broker to be set and solve this problem?

Upvotes: 0

Views: 1991

Answers (1)

Lior Chaga
Lior Chaga

Reputation: 1480

It really depends on the use case, you didn't really provide much information about the architecture. For instance - once the consumer is up, is it a long running consumer, or does it just wake up for a short while and consumes new messages arriving?

You can take any of the following approaches:

  • Filter ConsumerRecord by timestamp, so you will automatically throw away messages that were produced over configurable time.
  • In my team we're using ephemeral groups. That is - each time the service goes up, we generate a new group id for the consumer group, setting auto.offset.reset to latest
  • Seek to timestamp - since kafka 0.10 you can seek to a certain position. Use consumer.offsetsForTimes to get the offset of each topic partition for the desired time, and then use consumer.seek to get to the given offset.
  • If you use a consumer group, but never commit to kafka, then each time the a consumer is assigned to a topic partition, it will start consuming according to auto.offset.reset policy...

Upvotes: 1

Related Questions