CODEWITHSUNDEEP
apache-spark
adventure10
adventure10

Reputation: 13

Spark reading file from local and writing in hdfs

I have one file in local system. I want to read it using Spark locally and then write it in HDFS using the same spark program Is it possible ?

Upvotes: 1

Views: 4259

Answers (1)

s.polam
s.polam

Reputation: 10372

Client Mode If you run spark in client mode, Your driver will be running in your local system, so it can easily access your local files & write to HDFS.

In this mode to access your local files try appending your path after file://<local_path_file>

Cluster Mode If you run spark in cluster mode your driver will be launched from one of the worker, hence you can't access your local files in cluster mode.

spark.read.text("file:///tmp/srinivas/hive.log") // To access local files, without file:// It will search in HDFS location.

Upvotes: 3

Related Questions