CODEWITHSUNDEEP
apache-spark
Yohan Liyanage
Yohan Liyanage

Reputation: 7000

Apache Spark custom log4j configuration for application

I would like to customize the Log4J configuration for my application in a standalone Spark cluster. I have a log4j.xml file which is inside my application JAR. What is the correct way to get Spark to use that configuration instead of its own Log4J configuration?

I tried using the --conf options to set the following, but no luck.

spark.executor.extraJavaOptions -> -Dlog4j.configuration=log4j.xml
spark.driver.extraJavaOptions -> -Dlog4j.configuration=log4j.xml

I am using Spark 1.4.1 and there's no log4j.properties file in my /conf.

Upvotes: 5

Views: 4245

Answers (2)

Patrick McGloin
Patrick McGloin

Reputation: 2234

Try using driver-java-options. For example:

spark-submit --class my.class --master spark://myhost:7077 --driver-java-options "-Dlog4j.configuration=file:///opt/apps/conf/my.log4j.properties" my.jar

Upvotes: 3

mehmetminanc
mehmetminanc

Reputation: 1379

If you are using SBT as package manager/builder:

There is a log4j.properties.template in $SPARK_HOME/conf

  • copy it in your SBT project's src/main/resource
  • remove the .template suffix
  • edit it to fit your needs
  • SBT run/package/* will include this in JAR and Spark references it.

Works for me, and will probably include similar steps for other package managers, e.g. maven.

Upvotes: 6

Related Questions