CODEWITHSUNDEEP
hadoopapache-sparkapache-spark-sql
OneWorld
OneWorld

Reputation: 992

Spark - java IOException :Failed to create local dir in /tmp/blockmgr*

I was trying to run a long running Spark Job. After few hours of execution, I get exception below :

Caused by: java.io.IOException: Failed to create local dir in /tmp/blockmgr-bb765fd4-361f-4ee4-a6ef-adc547d8d838/28

Tried to get around it by checking:

  1. Permission issue in /tmp dir. The spark server is not running as root. but /tmp dir should be writable to all users.

  2. /tmp Dir has enough space.

Upvotes: 19

Views: 31393

Answers (3)

Mohan Kumar Kannan
Mohan Kumar Kannan

Reputation: 21

After verifying all the permissions and user access.

I got the same issue when building the components in Talend studio and it resolved by providing the correct "/" in spark scratch directory (temp directory) in spark Configuration tab. This is required when building the jar in windows and running in Linux cluster.

Upvotes: 0

Ladislav Vrbsky
Ladislav Vrbsky

Reputation: 51

Edit: The answer below did not eventually solve my case. It's because some subfolders spark (or some of its dependencies) was able to create, yet not all of them. The frequent necessity of creation of such paths would make any project unviable. Therefore I ran Spark (PySpark in my case) as an Administrator, which solved the case. So in the end it is probably a permission issue afterall.

Original answer:
I solved the same problem I had on my local Windows machine (not a cluster). Since there was no problem with permissions, I created the dir that Spark was failing to create, i.e. I created the following folder as a local user and did not need to change any permissions on that folder. 

C:\Users\<username>\AppData\Local\Temp\blockmgr-97439a5f-45b0-4257-a773-2b7650d17142

Upvotes: 3

Yaron
Yaron

Reputation: 10450

Assuming that you are working with several nodes, you'll need to check every node participate in the spark operation (master/driver + slaves/nodes/workers).

Please confirm that each worker/node have enough disk space (especially check /tmp folder), and right permissions.

Upvotes: 4

Related Questions