CODEWITHSUNDEEP
apache-spark
Knows Not Much
Knows Not Much

Reputation: 31546

Spark Target log file already exists

I am running a spark job. I can see that my job always fails and in the history server I see error

java.io.IOException: Target log file already exists (hdfs://nameservice1/user/spark/applicationHistory/application_1456200816465_392022)
    at org.apache.spark.scheduler.EventLoggingListener.stop(EventLoggingListener.scala:201)
    at org.apache.spark.SparkContext$$anonfun$stop$5.apply(SparkContext.scala:1394)
    at org.apache.spark.SparkContext$$anonfun$stop$5.apply(SparkContext.scala:1394)
    at scala.Option.foreach(Option.scala:236)
    at org.apache.spark.SparkContext.stop(SparkContext.scala:1394)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:107)
    at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

This is very confusing because it doesn't even enter my code.... I have no idea what is going wrong here. I tried deleting the directory and running again... but same issue.

Upvotes: 1

Views: 1702

Answers (2)

valiano
valiano

Reputation: 18551

I encountered a similar error when running Spark in standalone mode (local cluster) using spark-shell.

It occurred due to misconfiguration - I had both SPARK_LOG_DIR and SPARK_WORKER_DIR pointing to the same folder, with spark.eventLog.enabled=true for enabling event logs.

The worker tried to create a folder /var/log/spark/app-YYYYMMDDHHMMSS-000N containing a numbered folder per executor. In the same time Spark tried to create the event log for the application in a text file with the same name, so creating the file failed with IoException: target log file already exists.

The solution was adding /work to the Spark log location for seperating between the worker logs and the event logs.

Upvotes: 0

vanza
vanza

Reputation: 9903

You're running into https://issues.apache.org/jira/browse/SPARK-4705, which has been fixed for a while now. But it's just a symptom of a larger issue - your application is failing for some reason, and when YARN retries it, you get that error.

Upvotes: 3

Related Questions