hadoop log doesn't have complete info

Question

I am using hadoop apache 2.7.1 cluster which consists of 4 data nodes and two name nodes becuase it is high available

deployed on centos 7
and it began working at 01-08-2017

and we know that logs will be generated for each service

and let's take the current logs for example

hadoop-root-datanode-dn1.log 
hadoop-root-datanode-dn2.log 

where hadoop_root is the user iam logging with

my problem is:
in dn1 log i can find info from 01-08-2017 until today but in dn2 log doesn't have all complete info ,as it is emptied every day so it has only info related to today

is there any properties to control this behavior or it is centos problem
any help please ?

Answer 1

By default, the .log files are rotated daily by log4j. This is configurable with /etc/hadoop/conf/log4j.properties.

https://blog.cloudera.com/blog/2009/09/apache-hadoop-log-files-where-to-find-them-in-cdh-and-what-info-they-contain/

Not to suggest you're running a Cloudera cluster, but if you did, those files are not deleted. They're rolled and renamed

Oh, and I would suggest not running your daemons as root. Most hadoop installation guides explicitly have you create a hdfs or hadoop user