Kafka - Broker fails because all log dirs have failed

Question

I am attempting a simple kafka config on Windows os. My zookeeper and kafka installations use default configs (except for data and log dir paths). I can start kafka and produce/consume messages without issue; however, when the broker attempts to delete old messages (I set log retention to 100 ms), I get the following error:

    java.nio.file.FileSystemException: C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log -> C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.

        at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
        at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
        at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
        at java.nio.file.Files.move(Files.java:1395)
        at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:697)
        at org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:212)
        at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:415)
        at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:1601)
        at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:1588)
        at kafka.log.Log$$anonfun$deleteSegments$1$$anonfun$apply$mcI$sp$1.apply(Log.scala:1170)
        at kafka.log.Log$$anonfun$deleteSegments$1$$anonfun$apply$mcI$sp$1.apply(Log.scala:1170)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at kafka.log.Log$$anonfun$deleteSegments$1.apply$mcI$sp(Log.scala:1170)
        at kafka.log.Log$$anonfun$deleteSegments$1.apply(Log.scala:1161)
        at kafka.log.Log$$anonfun$deleteSegments$1.apply(Log.scala:1161)
        at kafka.log.Log.maybeHandleIOException(Log.scala:1678)
        at kafka.log.Log.deleteSegments(Log.scala:1161)
        at kafka.log.Log.deleteOldSegments(Log.scala:1156)
        at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1228)
        at kafka.log.Log.deleteOldSegments(Log.scala:1222)
        at kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:854)
        at kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:852)
        at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
        at scala.collection.immutable.List.foreach(List.scala:392)
        at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
        at kafka.log.LogManager.cleanupLogs(LogManager.scala:852)
        at kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:385)
        at kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110)
        at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:62)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: java.nio.file.FileSystemException: C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log -> C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.

                at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
                at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
                at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
                at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
                at java.nio.file.Files.move(Files.java:1395)
                at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:694)
                ... 32 more

And then...

[2018-08-01 18:14:01,479] INFO [ReplicaManager broker=0] Stopping serving replicas in dir C:\Workspace\kafka_2.11-1.1.0\kafka-logs (kafka.server.ReplicaManager)
[2018-08-01 18:14:01,480] ERROR Uncaught exception in scheduled task 'kafka-log-retention' (kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.KafkaStorageException: Error while deleting segments for discrete-topic-0 in dir C:\Workspace\kafka_2.11-1.1.0\kafka-logs
Caused by: java.nio.file.FileSystemException: C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log -> C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.
[2018-08-01 18:14:01,504] INFO Stopping serving logs in dir C:\Workspace\kafka_2.11-1.1.0\kafka-logs (kafka.log.LogManager)
[2018-08-01 18:14:01,508] ERROR Shutdown broker because all log dirs in C:\Workspace\kafka_2.11-1.1.0\kafka-logs have failed (kafka.log.LogManager)

This seems to be this issue: https://issues.apache.org/jira/browse/KAFKA-6188

kafka version: kafka_2.11-1.1.0 Windows 10 Zookeeper version: 3.4.12

I have a few questions:

1. Has anyone seen this issue and is there a workaround?
2. Is running a kafka broker on Windows a viable options at this point? I know it is not supported for Windows and there seem to be multiple critical issues.

Thanks

Answer 1

Simple method that work for me is just delete the both folder(kafka-logs & zookeeper) in this directory : C:\tmp

see the image for better understanding

Answer 2

As all the answers have parts of correct points, but none of them was a complete solution for me, I will add what helped me:

1. Stop both - Kafka and ZooKeeper;
2. Locate Kafka logs directory (value of the log.dirs variable in KAFKA_HOME\config\server.properties file) and clean it;
3. Locate ZooKeeper data directory (value of the data.Dir variable in KAFKA_HOME\config\zookeeper.properties file) and clean it;
4. Restart ZooKeeper;
5. Restart Kafka.

Answer 3

chown kafka -R /home/kafka/kafkalogs

worked for me

Answer 4

You can delete all Kafka data. See Kafka docs:

1. Stop the producer and consumer clients with Ctrl-C, if you haven't done so already.
2. Stop the Kafka broker with Ctrl-C.
3. Lastly, stop the ZooKeeper server with Ctrl-C.

If you also want to delete any data from your local Kafka environment including any events you have created along the way, run the command:

$ rm -rf /tmp/kafka-logs /tmp/zookeeper

If you are on a Windows machine, you can use the following command

rmdir /s /q C:\tmp\kafka-logs

rmdir /s /q C:\tmp\zookeeper

Answer 5

Similar to what Tutor did above, what worked for me was deleting the zookeeper-data folder. I'm on Windows 10, so the zookeeper-data folder was at my root Kafka folder, and defined in config/zookeeper.properties

dataDir=c:/projects/kafka/zookeeper-data

FYI..you'll lose whatever topics, etc. you've already created

Answer 6

The problem is because of that you are running out of storage for logs.

Change log.dirs in server.properties. Also make sure that you and then restart Kafka.

P.S. 1: By doing so, the older messages will be lost.

P.S. 2: Don't set the tmp/ as log.dirs location, since it is purged once your machine turns off.

Answer 7

Remove all files under the local folder as shown in your docker monitor. After, restart the containers.

Answer 8

Try creating a new folder inside the Kafka folder. Change server.properties and zookeeper.properties and restart the command prompts and again try starting the servers.

Answer 9

In dev machine, I stopped zookeeper and kafka-server and all client and deleted the contents of the logs directory. Then started the zookeeper, kafka, connectors, clients in sequence.

Answer 10

Go to config folder in kafka folder (e.g. D:\kafka\config)  open zookeeper.properties Edit your datadir to look like : dataDir = D:/kafka/data/zookeeper open server.properties in notepad and edit logs.dirs and zookeeper.connect logs.dirs = D:/kafka/data/kafka zookeeper.connect=localhost:2181 NOTE: replace D:\kafka according to your own settings and folders you have for kafka

Answer 11

Delete the log files of Kafka brokers and also the data and log files of zookeeper and restart both. Also keep in mind to stop the Kafka clients before you restart the Kafka and Zookeeper

Answer 12

I tried multiple methods to solve this issue:

1. deleting logs from folder pointed by log.dirs property in server.properties. it does solve my problem for certain duration but it stoped working when i had more than 20 topics created so each time i start my kafka server this issue was coming.

2. tried deleting topics from kafka using command "kafka-topics --zookeeper localhost:2181 --delete --topic topicname" but it marked my topic as --"marked to be deleted flag". but did not solved my problem.

then finally i tried deleting folders zookeeper and kafka-logs inside tmp folder of kafka and it worked and all topics were deleted.

Answer 13

I am using Kafka 2.12-2.3.0 on Windows. And What worked for me was changing server.properties setting log.dirs= to contains both log data Kafka and Zookeeper.

Example :

log.dirs=C:/kafka_2.12-2.3.0/data/kafka, C:/kafka_2.12-2.3.0/data/zookeeper

Answer 14

I faced similar issue when I run zookeeper, kafka server, command line producer and consumer locally with default configs as described here.

Server and producer started successfully, but when I tried to start consumer I immediately got the issue with failed logs.

I fixed it by setting different log directories for server, producer and consumer. As far as there is no explicit way to set it for producer and consumer I had to update their start scripts. I added a line

set LOG_DIR=/distr/kafka_2.11-2.2.0/producer_logs

to kafka_console_producer.bat
And /distr/kafka_2.11-2.2.0/consumer_logs for kafka_console_consumer.bat respectively.
After the fix consumer successfully started. Seems the problem is because all three processes use the same log directory and interfere with each other somehow.

Answer 15

I had the same kafka behavior but simply renaming or cleaning log directories wouldn't do the trick. And every time I tried to start kakfa it would crash (it was generating back all the logs deleted).

Solution:

1. Delete the logs as before.
2. Go to ZooKeeper data dir (you find it in \conf\zoo.cfg) and clean it. Restart everything and it should be completely reset.

Answer 16

step 1 : point to a new log directory in server.properties file and save the file

log.dirs=C:\Tools\kafka_2.11-2.1.0\kafka-test-logs

step 2 : start the kafka server again

/bin/windows/kafka-server-start.bat /config/server.properties