CODEWITHSUNDEEP
loggingapache-sparklog4jhadoop-yarn
Sameeksha Raina
Sameeksha Raina

Reputation: 209

How to customize Log4j properties file in YARN

I am running an application on top of YARN in spark cluster but I need to set my own path where log files are created and need to change the formatting of log message. For this I want to use my customized Log4j.properties file instead of the default log4j.properties file provided by YARN. How can this be done?

Upvotes: 0

Views: 1888

Answers (1)

Brad
Brad

Reputation: 15879

I use this spark-submit script to push my log4j.properties up to Yarn so that I can control the logging pattern. I particularly wanted the full package names so I can grep out what my classes where logging, from the resulting Yarn application logs.

spark-submit \
    --class com.acme.Main \
    --master yarn \
    --deploy-mode cluster \
    --driver-memory 2g \
    --executor-memory 1g \
    --driver-class-path "./conf" \
    --files "./conf/app.properties,./conf/log4j.properties" \
    ./lib/my-app-uber.jar \
    "$@"

Personally I only use the ConsoleAppender and all my application logging appears in the standard Yarn logs along with everything else. You can't use the RollingFileAppender with HDFS.

log4j.rootLogger=INFO, stdout

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} - %-5p %c:%L -> %m%n

You should see your results in 

yarn logs -applicationId <your-app-id>

Upvotes: 1

Related Questions