CODEWITHSUNDEEP
apache-kafka
Midhun Mathew Sunny
Midhun Mathew Sunny

Reputation: 1361

Which directory does apache kafka store the data in broker nodes

I can see a property in config/server.properties called log.dir? Does this mean kafka uses the same directory for storing logs and data both?

Upvotes: 65

Views: 115632

Answers (3)

amitsahu07
amitsahu07

Reputation: 49

log.dir in server.properties is the place where the Kafka broker will store the commit logs containing your data. Typically this will your high speed mount disk for mission critical use-cases.

For application/broker logging you can use general log4j logging to get the event logging in your custom location. Below are the variables to do this.

-Dlog4j.configuration=file:<configuration file with log rolling, logging level etc.>  & -Dkafka.logs.dir=<path to logs>

Upvotes: 4

Matthias J. Sax
Matthias J. Sax

Reputation: 62350

Kafka topics are "distributed and partitioned append only logs". Parameter log.dir defines where topics (ie, data) is stored.

It is not related to application/broker logging.

The default log.dir is /tmp/kafka-logs which you may want to change in case your OS has a /tmp directory cleaner.

Upvotes: 94

amethystic
amethystic

Reputation: 7091

log.dir or log.dirs in the config/server.properties specifiy the directories in which the log data is kept. The server log directory is kafka_base_dir/logs by default. You could modify it by specifying another directory for 'kafka.logs.dir' in log4j.properties.

Upvotes: 18

Related Questions