CODEWITHSUNDEEP
apache-sparklog4jspark-streaminghadoop-yarn
Akshansh Rajput
Akshansh Rajput

Reputation: 63

Capture spark executor logs in local file on YARN CUSTER MODE

I am running spark streaming in yarn cluster mode and i want to capture logs and write it in driver local file for this I have created custom log4j.properties files in which I have mentioned driver's local file path but I can only see drivers logs in this file, Why my executors logs are not captured in this file and how can I capture executor log. I have tried different approaches and my spark-submit command is as follows:-

spark-submit --master yarn --deploy-mode yarn-cluster
--conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=file:/home/log/conf/log4j.properties"
--conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/home/log/conf/log4j.properties" --class com.Word.count.SparkStream /home/project/WordCount/target/Count-0.0.1-SNAPSHOT.jar

Upvotes: 1

Views: 577

Answers (1)

roamer
roamer

Reputation: 163

you may post your log4j.properties.I assume that you can see executor logs in executor node local dir

Upvotes: 0

Related Questions