CODEWITHSUNDEEP
apache-spark
user1870400
user1870400

Reputation: 6364

Spark Jobs filling up the disk at SPARK_LOCAL_DIRS location

spark jobs are filling up the disk within a short amount of time (< 10 mins). I have a disk space of 10GB and it is getting full at SPARK_LOCAL_DIRS location. In my case SPARK_LOCAL_DIRS is set to /usr/local/spark/temp.

There are lot of files like this input-0-1489072623600 and each file is somewhere between 3MB-8MB.

any ideas?

Upvotes: 0

Views: 1252

Answers (1)

hemant singh
hemant singh

Reputation: 11

SPARK_LOCAL_DIRS is used for rdd cache(disk) and shuffle data. You should check the storage details whether how much data is cached(disk) and if any shuffle operations during your job.

Upvotes: 1

Related Questions