CODEWITHSUNDEEP
apache-spark
msharky
msharky

Reputation: 129

Configuring master node in spark cluster

Apologies in advance as I am new to spark. I have created a spark cluster in standalone mode with 4 workers, and after successfully being able to configure worker properties, I wanted to know how to configure the master properties.

I am writing an application and connecting it to the cluster using SparkSession.builder (I do not want to submit it using spark-submit.)

I know that that the workers can be configured in the conf/spark-env.sh file and has parameters which can be set such as 'SPARK_WORKER_MEMORY' and 'SPARK_WORKER_CORES'

My question is: How do I configure the properties for the master? Because there is no 'SPARK_MASTER_CORES' or 'SPARK_MASTER_MEMORY' in this file.

I thought about setting this in the spark-defaults.conf file, however it seems that this is only used for spark-submit.

I thought about setting it in the application using SparkConf().set("spark.driver.cores", "XX") however this only specifies the number of cores for this application to use.

Any help would be greatly appreciated.

Thanks.

Upvotes: 1

Views: 1488

Answers (1)

Abdulrahman
Abdulrahman

Reputation: 453

Three ways of setting the configurations of Spark Master node (Driver) and spark worker nodes. I will show examples of setting the memory of the master node. Other settings can be found here

1- Programatically through SpackConf class.

Example: 

new SparkConf().set("spark.driver.memory","8g")

2- Using Spark-Submit: make sure not to set the same configuraiton in your code (Programatically like 1) and while doing spark submit. if you already configured settings programatically, every job configuration mentioned in spark-submit that overlap with (1) will be ignored.

example :

spark-submit --driver-memory 8g

3- through the Spark-defaults.conf: In case none of the above is set this settings will be the defaults.

example :

spark.driver.memory     8g

Upvotes: 1

Related Questions