CODEWITHSUNDEEP
apache-spark
Pyd
Pyd

Reputation: 6159

How to pass a local file as input in spark-submit

How to pass a local file as input in spark-submit, I have tried like below:

spark-submit --jars /home/hduser/.ivy2/cache/com.typesafe/config/bundles/config-1.3.1.jar --class "retail.DataValidator" --master local[2] --executor-memory 2g --total-executor-cores 2 sample-spark-180417_2.11-1.0.jar file:///home/hduser/Downloads/Big_Data_Backup/ dev file:///home/hduser/spark-training/workspace/demos/output/destination file:///home/hduser/spark-training/workspace/demos/output/extrasrc file:///home/hduser/spark-training/workspace/demos/output/extradest

Error:

Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: file:/home/inputfile , expected: hdfs://hadoop:54310

also tried path without the prefix "file://", but no luck. Its working fine in eclipse.

Thank you.!

Upvotes: 0

Views: 4184

Answers (1)

Piotr Kalański
Piotr Kalański

Reputation: 689

If you want those files be accessible by each executor you need to use option files. Example:

spark-submit --files file1,file2,file3

Upvotes: 1

Related Questions