CODEWITHSUNDEEP
apache-spark
Dmitri
Dmitri

Reputation: 153

Specify spark driver for spark-submit

I'm submitting a spark job from a shell script that has a bunch of env vars and parameters to pass to spark. Strangely, the driver host is not one of these parameters (there are driver cores and memory however). So if I have 3 machines in the cluster, a driver will be chosen randomly. I don't want this behaviour since 1) the jar I'm submitting is only on one of the machines and 2) the driver machine should often be smaller than the other machines which is not the case if it's random choice.

So far, I found no way to specify this param on the command line to spark-submit. I've tried --conf SPARK_DRIVER_HOST="172.30.1.123, --conf spark.driver.host="172.30.1.123 and many other things but nothing has any effect. I'm using spark 2.1.0. Thanks.

Upvotes: 1

Views: 2663

Answers (2)

Paul Velthuis
Paul Velthuis

Reputation: 335

From the documentation: http://spark.apache.org/docs/latest/running-on-yarn.html

When running the cluster in standalone or in Mesos the driver host (this is the master) can be launched with:

--master <master-url> #e.g. spark://23.195.26.187:7077

When using YARN it works a little different. Here the parameter is yarn

--master yarn

The yarn is specified in Hadoop its configuration for the ResourceManager. For how to do this see this interesting guide https://dqydj.com/raspberry-pi-hadoop-cluster-apache-spark-yarn/ . Basically in the hdfs the hdfs-site.xml and in yarn the yarn-site.xml

Upvotes: 1

FaigB
FaigB

Reputation: 2281

I assume you are running with Yarn cluster. In brief yarn uses containers to launch and implement tasks. And resource manager decides where to run which container based on availability of resources. In spark case drivers and executors also launched as containers with separate jvms. Driver dedicated to splitting tasks among executors and collect the results from them. If your node from where you launch your application included in cluster then it will be also used as shared resource for launching driver/executor.

Upvotes: 1

Related Questions