CODEWITHSUNDEEP
apache-spark
hawarden_
hawarden_

Reputation: 2170

Spark: which directory spark-submit will take to find --files?

I saw that we can use spark-submit --files to add files to the job. It's not a problem if I specify the absolute path like spark-submit --files /etc/somescript.sh.

But what directory it will search for if I just put spark-submit --files somescript.sh ? Is it the current workplace folder or all the classpath?

Upvotes: 1

Views: 932

Answers (1)

Mohana B C
Mohana B C

Reputation: 5487

Spark will search the specified file in the current folder.

You can check the code flow on spark's github repo.

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L402

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L2069

Upvotes: 1

Related Questions