CODEWITHSUNDEEP
mongodbapache-kafkaapache-kafka-connectconfluent-platformdebezium
Nil
Nil

Reputation: 125

Kafka brokers shuts down because log dirs have failed

I have a 3 broker Kafka clusters with the Kafka logs in the /tmp directory. I am running Debezium Source Connector to MongoDB which polls data from 4 collections.

However within 5 mins after starting the connector, the Kafka brokers are shutting down with the following error:

[2020-04-16 18:25:08,642] ERROR Shutdown broker because all log dirs in /tmp/kafka-logs-1 have failed (kafka.log.LogManager)

I have tried the different suggestions viz. Deleting the Kafka logs and cleaning out the Zookeeper logs. But I ran into the same problem again.

I have also noticed that the kafka logs occupy 100% of the /tmp directory when this happens. So I have also changed the log retention policy based on size.

log.retention.hours=168
log.retention.bytes=1073741824
log.segment.bytes=1073741824
log.retention.check.interval.ms=10000

This also turned up to be futile. I would like to have some assistance regarding this. Thanks in advance!

Upvotes: 0

Views: 4693

Answers (1)

Giorgos Myrianthous
Giorgos Myrianthous

Reputation: 39790

Your log files got corrupted probably because you've ran out of storage.

I would suggest to change log.dirs in server.properties. Also make sure that you don't use the tmp/ location, as this is going to be purged once your machine turns off. Once you have changed log.dirs you can restart Kafka.

Note that the older messages will be lost.

Upvotes: 2

Related Questions