CODEWITHSUNDEEP
apache-kafka
Knight71
Knight71

Reputation: 2949

log retention bytes per topic not working with kafka

I tried setting the log.retention.bytes per topic by using the below command

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic app.test.client --config retention.bytes=10485760

Verified the config setting by doing a

bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic 
app.test.client 
Topic:app.test.client   PartitionCount:3     ReplicationFactor:2     Configs:retention.bytes=10485760
    Topic: app.test.client  Partition: 0    Leader: 2       Replicas: 2,1   Isr: 2,1
    Topic: app.test.client  Partition: 1    Leader: 0       Replicas: 0,2   Isr: 0,2
    Topic: app.test.client  Partition: 2    Leader: 1       Replicas: 1,0   Isr: 1,0

I have set my retention bytes to 10MB -> 10485760. But my log files were growing even after 10MB. 

$du -hs /mnt/kafka/logs/*
128M    /mnt/kafka/logs/app.test.client-1
128M    /mnt/kafka/logs/app.test.client-2

Version Kafka - 0.8.2.1

How to properly set this setting per topic ?

Upvotes: 3

Views: 5255

Answers (2)

Hanan Oanunu
Hanan Oanunu

Reputation: 179

Kafka retention is made on a segment level. Meaning, when Kafka exceeds the retention boundaries (time or size) it will look for the segments to delete, in order to get back in the retention boundaries. In cases where retention.bytes is smaller than segment.bytes, you may hit a situation where the is only one segment, which is also the current used segment, so deletion process may work not exactly as expected.

Try to set segment.bytes to a value smaller than retention.bytes.

Upvotes: 5

Marko Bonaci
Marko Bonaci

Reputation: 5708

I think that log.retention.bytes doesn't impose a hard limit on a topic/partition size, but rather signal to Kafka when to delete the oldest segment file(s) that are above the value you set this property to.

Upvotes: 2

Related Questions