CODEWITHSUNDEEP
apache-kafka
zubaazu gk
zubaazu gk

Reputation: 23

Apache Kafka Log retention configuration issues or How to configure the Kafka retention policy?

I am using kafka_2.12-1.1.0. Below are my retention conf. I am using Win OS.

log.retention.ms=120000

log.segment.bytes=2800

log.retention.check.interval.ms=30000

I see the logs get rolled over and generates *0000.log to *0069.log and then *00103.log etc and then i expected it to delete the old files but when the retention logic kicks in I get below error process cannot access the file because it is being used by another process and the kafka server crashes and doesnt recover from there. This happens when it is trying to delete 00000000000000000000.log ? Can you let me know the commands to check if they are active segments or why is it trying to delete this and crashing.

PFB log file if you want to see it.

[ProducerStateManager partition=myLocalTopic-0] Writing producer snapshot at offset 39

[Log partition=myLocalTopic-0, dir=D:\tmp\kafka-logs] Rolled new log segment at offset 39 in 39 ms. 

[ProducerStateManager partition=myLocalTopic-0] Writing producer snapshot at offset 69 

[Log partition=myLocalTopic-0, dir=D:\tmp\kafka-logs] Rolled new log segment at offset 69 in 52 ms.

[ProducerStateManager partition=myLocalTopic-0] Writing producer snapshot at offset 91

[Log partition=myLocalTopic-0, dir=D:\tmp\kafka-logs] Rolled new log segment at offset 91 in 18 ms. 

[Log partition=myLocalTopic-0, dir=D:\tmp\kafka-logs] Found deletable segments with base offsets [0,69] due to retention time 120000ms breach (kafka.log.Log)

[ProducerStateManager partition=myLocalTopic-0] Writing producer snapshot at offset 103 (kafka.log.ProducerStateManager)

[Log partition=myLocalTopic-0, dir=D:\tmp\kafka-logs] Rolled new log segment at offset 103 in 25 ms. (kafka.log.Log)

 [Log partition=myLocalTopic-0, dir=D:\tmp\kafka-logs] Scheduling log segment [baseOffset 0, size 2746] for deletion. (kafka.log.Log)

ERROR Error while deleting segments for myLocalTopic-0 in dir D:\tmp\kafka-logs (kafka.server.LogDirFailureChannel)
java.nio.file.FileSystemException: D:\tmp\kafka-logs\myLocalTopic-0\00000000000000000000.log -> D:\tmp\kafka-logs\myLocalTopic-0\00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.
.............   at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:98)
.......................................................
 ERROR Uncaught exception in scheduled task 'kafka-log-retention' (kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.KafkaStorageException: Error while deleting segments for myLocalTopic-0 in dir D:\tmp\kafka-logs
Caused by: java.nio.file.FileSystemException: D:\tmp\kafka-logs\myLocalTopic-0\00000000000000000000.log -> D:\tmp\kafka-logs\myLocalTopic-0\00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.

Click here for file list image

Upvotes: 1

Views: 3516

Answers (1)

Jayz
Jayz

Reputation: 654

Apache Kafka provides us with the following retention policies

  1. Time based Retention
  2. Size based Retention

For time based retention try the following commands,

Command to set retention time

./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic --config retention.ms=1680000

Command to remove Topic level retention time 

./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic  --delete-config retention.ms

For size based retention try the following commands, To set retention size:

./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic --config retention.bytes=104857600

To remove,

./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic  --delete-config retention.bytes

Upvotes: 1

Related Questions