CODEWITHSUNDEEP
apache-kafka
Kevin Heirich
Kevin Heirich

Reputation: 119

Kafka retention policy not working as expected

I would like to delete some records after some time. For testing purpose I've used the command

bin/kafka-configs.sh --zookeeper localhost:2181 --alter --entity-type topics --entity-name my-topic-name --add-config retention.ms=1

Which in my understanding should purge some records (or at least make them disappear when a consumer start reading messages from the beginning) after 1 milisecond.

But what actually happened is weird. Every about 5 minutes, the last 5 minutes messages are deleted and do not appear anymore. so for instance if I send a message every minute, I would have 5 messages, then 0, the 5 again, then 0, etc.

So I guess my command isn't working at all ?

I use kafka 2.8.0

Many thanks ! Kev

Upvotes: 0

Views: 5196

Answers (1)

Sahil Gupta
Sahil Gupta

Reputation: 2166

  1. 'retention.ms' property will work when log.cleanup.policy = 'delete'

    • This should be set at your end, i.e why data is getting cleaned up

  2. 'log.retention.check.interval.ms' should be less then retention.ms if you want your data to be deleted after every 'retention.ms' milliseconds.

enter image description here Screenshot from official-doc

Since the default value of 'log.retention.check.interval.ms' is 5 min, hence messages are deleted after every 5 minutes.

Upvotes: 3

Related Questions