CODEWITHSUNDEEP
apache-spark
xv70
xv70

Reputation: 982

Where is Spark writing SaveAsTextFile in cluster?

I'm a bit at loss here (Spark newbie). I spun up an EC2 cluster, and submitted a Spark job which saves as text file in the last step. The code reads 

reduce_tuples.saveAsTextFile('september_2015')

and the working directory of the python file I'm submitting is /root. I cannot find the directory called september_2005, and if I try to run the job again I get the error:

: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://ec2-54-172-88-52.compute-1.amazonaws.com:9000/user/root/september_2015 already exists

The ec2 address is the master node where I'm ssh'ing to, but I don't have a folder /user/root.

Seems like Spark is creating the september_2015 directory somehwere, but find doesn't find it. Where does Spark write the resulting directory? Why is it pointing me to a directory that doesn't exist in the master node filesystem?

Upvotes: 1

Views: 1051

Answers (1)

Reactormonk
Reactormonk

Reputation: 21740

You're not saving it in the local file system, you're saving it in the hdfs cluster. Try eph*-hdfs/bin/hadoop fs -ls /, then you should see your file. See eph*-hdfs/bin/hadoop help for more commands, eg. -copyToLocal.

Upvotes: 2

Related Questions