firemonkey
Reputation: 329
apache spark saveAsObjectFile writes to hdfs by default
When I am running spark locally(non hdfs), RDD saveAsObjectFile writes the file to local file system (ex : path /data/temp.txt)
when I am running spark on YARN cluster, RDD saveAsObjectFile writes the file to hdfs. (ex : path /data/temp.txt )
Is there a way to explictly mention local file system instead of hdfs when running spark on YARN cluster.
Upvotes: 1
Views: 2004
Answers (1)
suztomo
Reputation: 5202
You can explicitly specify "file:///" prefix in the argument.
yourRDD. saveAsObjectFile("file:///path/to/local/filesystem")
Upvotes: 7
Related Questions
- Spark failing HDFS writes
- How to write to HDFS from a Spark Streaming script
- How spark writes to HBASE
- HDFS Write Issue in Kerberos in Spark YARN Application
- Spark won't run final `saveAsNewAPIHadoopFile` method in yarn-cluster mode
- Why does DataFrame.saveAsTable("df") save table to different HDFS host?
- Spark streaming output not saved to HDFS file
- Save image file to HDFS using Spark
- Writing file to HDFS using Java
- Persist an entity object to HDFS using spark