CODEWITHSUNDEEP
apache-spark
user3861444
user3861444

Reputation:

Separate logs from Apache spark

I would like to have separate log files from workers, masters and jobs(executors, submits, don't know how call it). I tried configuration in log4j.properties like 

log4j.appender.myAppender.File=/some/log/dir/${log4j.myAppender.FileName}

and than passing log4j.myAppender.FileName in SPARK_MASTER_OPTS, SPARK_WORKER_OPTS, spark.executor.extraJavaOptions and spark.driver.extraJavaOptions.

It works perfectly well with workers and masters but fails with executors and drivers. Here is example of how I use these:

./spark-submit ... --conf "\"spark.executor.extraJavaOptions=log4j.myAppender.FileName=myFileName some.other.option=foo\"" ...

I also tried putting log4j.myAppender.FileName with some default value in spark-defaults.conf but it doesn't work neither.

Is there some way to achieve what I want?

Upvotes: 2

Views: 3234

Answers (1)

Brad
Brad

Reputation: 15879

Logging for Executors and Drivers can be configured by conf/spark-defaults.conf by adding these entries (from my windows config)

spark.driver.extraJavaOptions  -Dlog4j.configuration=file:C:/dev/programs/spark-1.2.0/conf/log4j-driver.properties
spark.executor.extraJavaOptions  -Dlog4j.configuration=file:C:/dev/programs/spark-1.2.0/conf/log4j-executor.properties

Note that each entry above references a different log4j.properties file so you can configure them independently.

Upvotes: 2

Related Questions