CODEWITHSUNDEEP
apache-sparkgoogle-cloud-dataproc
Girish
Girish

Reputation: 1717

Google cloud dataproc --files is not working

I want to copy some property files to master and workers while submitting spark job, so as stated in the doc I am using --files to copy the files on executors working directory. but below command is not copying anything in executors working directory. So anybody have idea please share.

gcloud dataproc jobs submit spark --cluster=cluster-name --class=dataproc.codelab.word_count.WordCount --jars=gs://my.jar --region=us-central1 --files=gs://my.properties -- gs://my/input/ gs://my/output3/

Upvotes: 2

Views: 1311

Answers (1)

Henry Gong
Henry Gong

Reputation: 316

According to official Spark documentation, when Spark is running on Yarn, the Spark executor will use local directory configured for Yarn as working directory, which is by default - /hadoop/yarn/nm-local-dir/usercache/{userName}/appcache/{applicationId}.

So based on you description if it dose show up there then it's working as expected.

Upvotes: 2

Related Questions