CODEWITHSUNDEEP
apache-sparkpysparkgoogle-cloud-dataprocspark-submitdataproc
figs_and_nuts
figs_and_nuts

Reputation: 5771

How to pass env variables in dataproc submit command?

I want to be able to set the following env variables while submitting a job via dataproc submit:

  1. SPARK_HOME
  2. PYSPARK_PYTHON
  3. SPARK_CONF_DIR
  4. HADOOP_CONF_DIR

How can I achieve that?

Upvotes: 2

Views: 1639

Answers (1)

Dagang Wei
Dagang Wei

Reputation: 26548

Check the doc Setting environment variables on Dataproc cluster nodes on how to set env variables for different components in Dataproc.

Upvotes: 1

Related Questions