CODEWITHSUNDEEP
apache-sparkhiveapache-spark-sql
Terry Dactyl
Terry Dactyl

Reputation: 1868

Spark staging directory race condition when writing to hive partition?

I am seeing intermittent exceptions when attempting to write a Dataset to a partition in a hive table.

Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: /user/hive/warehouse/devl_fr9.db/fr9_ftdelivery_cpy_2_4d8eebd3_9691_47ce_8acc_b2a5123dabf6/.spark-staging-d996755c-eb81-4362-a393-31e8387104f0/date_id=20180604/part-00000-d996755c-eb81-4362-a393-31e8387104f0.c000.snappy.parquet for client 10.56.219.20 already exists

If I check HDFS the relevant path does not exist. I can only assume this is some race condition regarding temp staging files. I am using Spark 2.3

Upvotes: 5

Views: 1702

Answers (1)

jmng
jmng

Reputation: 2568

A possible reason for this issue is that, during a job's execution, a task started writing data to that file and failed.

When a task fails, the data that it had already written is not deleted/purged by Spark (confirmed at least in 2.3 and 2.4). Therefore, when a different executor attempts to re-execute the failed task it will attempt to write to a file with the same name, and you'll get a FileAlreadyExistsException.

In your case, the file that already exists is called part-00000-d996755c-eb81-4362-a393-31e8387104f0.c000, so it's likely that you have a log message in stderr indicating that task 00000 was lost due to failure, something like 

WARN TaskSetManager: Lost task **00000** in stage...

If you fix the reason for this failure - probably an OutOfMemoryError, if the issue is intermitent - the FileAlreadyExistsException will likely be solved because the task will not fail and leave temporary files behind.

Upvotes: 1

Related Questions