CODEWITHSUNDEEP
hadoopapache-spark
Nikhil Utane
Nikhil Utane

Reputation: 1221

Spark: saveAsTextFile() only creating SUCCESS file and no part file when writing to local filesystem

I am writing an RDD to a file using below command:

rdd.coalesce(1).saveAsTextFile(FilePath)

When the FilePath is HDFS path (hdfs://node:9000/folder/) everything works fine.

When the FilePath is local path (file:///home/user/folder/) everything seems to work. The output folder is created and SUCCESS file is also present.

However I do not see any part-00000 file containing the output. There is no other file. There is no error in the spark console output either.

I also tried calling collect on the RDD before calling saveAsTextFile(), giving 777 permission to output folder but nothing is working.

Please help.

Upvotes: 10

Views: 2826

Answers (2)

OneCricketeer
OneCricketeer

Reputation: 191681

In order to save a Spark object to the local driver filesystem, you'll need to use collect(), then open a file yourself to write that collection into.

Otherwise, if you ran as part of a YARN job, for example, you should go look at the local filesystems of the nodemanagers where the Spark job ran

Upvotes: 0

dpwang
dpwang

Reputation: 81

save to local make effects only when using local master

Upvotes: 1

Related Questions