CODEWITHSUNDEEP
apache-kafka
DrGenius
DrGenius

Reputation: 967

Why Kafka stops with an error on log dirs?

I am trying to learn kafka and I had the below error:

[2021-01-21 13:46:43,247] WARN [ReplicaManager broker=0] Broker 0 stopped fetcher for partitions
__consumer_offsets-22,first_topic-2,__consumer_offsets-37,first_topic-0,__consumer_offsets- 
38,__consumer_offsets-13,twitter_tweets-5,__consumer_offsets-30,twitter_tweets-3,__consumer_offsets-
8,__consumer_offsets-21,__consumer_offsets-4,__consumer_offsets-27,__consumer_offsets- 
7,__consumer_offsets-9,__consumer_offsets-46,new_topic-0,__consumer_offsets-25,__consumer_offsets- 
35,twitter_tweets-0,__consumer_offsets-41,__consumer_offsets-33,__consumer_offsets- 
23,__consumer_offsets-49,__consumer_offsets-47,__consumer_offsets-16,__consumer_offsets- 
32,__consumer_offsets-40 and stopped moving logs for partitions  because they are in the failed log 
directory C:\kafka_2.13-2.6.0\data\kafka. (kafka.server.ReplicaManager)
[2021-01-21 13:46:43,252] WARN Stopping serving logs in dir C:\kafka_2.13-2.6.0\data\kafka 
(kafka.log.LogManager)
[2021-01-21 13:46:43,254] ERROR Shutdown broker because all log dirs in C:\kafka_2.13-2.6.0\data\kafka have failed (kafka.log.LogManager)

This happens every time I run a command for example:

bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic streams-plaintext-input.

When I delete all the offsets in /data folder then everything run smoothly. Is this happening because of the 7 day existing period that Kafka has?

Upvotes: 0

Views: 1008

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191681

Main issue is that Kafka depends on POSIX filesystem semantics that don't work well on windows.

Kafka uses specific features of POSIX to achieve high performance, so emulations—which happen on WSL 1—are insufficient. For example, the broker will crash when it rolls a segment file

This appears to be the error you're mentioning about the segment retention

If you want to use Kafka on windows, WSL2 is the suggested solution.

https://www.confluent.io/blog/set-up-and-run-kafka-on-windows-linux-wsl-2/

Also note: --zookeeper flag is deprecated

Upvotes: 1

Related Questions