CODEWITHSUNDEEP
apache-kafka
YVS1997
YVS1997

Reputation: 680

Is it safe to delete Kafka/log old file?

I have Kafka server instance in my cloud server, consist of 3 brokers 3 zk and 12 connect workers. As we know, Kafka main folder are consist of bin,config,and logs. I want to ask if i want to safe the disk size, is it safe to delete old server.log, connect.log and zookeeper_gc.log file in logs folder? And if it yes, then how i delete/compress it periodically (for example every day or week)?

Upvotes: 0

Views: 4141

Answers (2)

kazzaki
kazzaki

Reputation: 72

To answer your first question, Yes it is ok to delete old Kafka log files. Those are meant for your use only if you want to trace back the history logs.

For the second one, you can check the log4j properties file under /config folder. By default Kafka provides DailyRollingAppender which you can change to RollingAppender of your liking..

Snippets from default kafka log4j properties. Ref https://github.com/apache/kafka/blob/trunk/config/log4j.properties

log4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.kafkaAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.kafkaAppender.File=${kafka.logs.dir}/server.log
log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

You can potentially update the RollingAppender like below which will keep maximum 10 files of 25MB each so never occupying more than 250MB.

log4j.appender.kafkaAppender=org.apache.log4j.RollingFileAppender
log4j.appender.kafkaAppender.File=${kafka.logs.dir}/server.log
log4j.appender.kafkaAppender.MaxFileSize=25MB
log4j.appender.kafkaAppender.MaxBackupIndex=10
log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

Please note there are several appender in the above log properties file (ref above link), you got to update settings for each of those accordingly.

Upvotes: 4

khasim umer
khasim umer

Reputation: 21

Kafka uses log.cleanup.policy configuration property to define cleanup strategies. Furthermore, it uses log.retention.check.interval.ms configuration property as the interval between regular log checks. So the above two properties will allow one to cleanup old logs at required intervals.

Here are further details for your reference: https://jaceklaskowski.gitbooks.io/apache-kafka/content/kafka-log-cleanup-policies.html

Upvotes: 0

Related Questions