CODEWITHSUNDEEP
apache-kafka
Manish Mishra
Manish Mishra

Reputation: 848

Dynamically configuring retention.ms not working for a kafka topic

I have a Kafka topic called retention and Below are the server configuration related to retention: 

log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=3600000 (~ 1 hour)
log.cleaner.enable=true

And below is the topic specific config: 

retention.ms=2592000000,retention.bytes=3298534883328

where retention.ms ~ 30d and retention.bytes = ~ 3.29 TB

I configured the retention.ms and retention.bytes using the below command recently (on 14th Jan 2019) using below commands: 


./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic retentions --config retention.bytes=219902325555

Here the configuration for the retntion.bytes seems to be working while retention.ms does not seem to be working. Here is the evidence that I could collect: 

cd log_dir/retentions-0/
ls -lrt 00000000000000000000.*
-rw-r--r-- 1 root root 294387381 Nov 26 22:37 00000000000000000000.log
-rw-r--r-- 1 root root      3912 Jan 14 18:06 00000000000000000000.index
-rw-r--r-- 1 root root      5868 Jan 14 18:06 00000000000000000000.timeindex

If we look into the logs of older segments these are nearly 2 months old.

Can anybody tell which of these two configurations will take effect on priority Or, both can work whichever crosses the configured threshold.

In my assumptions, both configurations should work in conjunction. Plz, let me know if this is not the case.

Upvotes: 0

Views: 1094

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191710

Both work in conjunction.

From Kafka: The Definitive Guide book

If you have specified a value for both log.retention.bytes and log.retention.ms ... messages may be removed when either criteria is met.

Upvotes: 1

Related Questions