CODEWITHSUNDEEP
apache-spark
Oscar
Oscar

Reputation: 855

Spark Configuration: memory/instance/cores

Lately I have found myself a bit confused between the different SPARK settings spark.executor.memory, SPARK_WORKER_MEMORY, SPARK_MEM, SPARK_MASTER_MEMORY, and the relationship to SPARK_WORKER_INSTANCES and SPARK_WORKER_CORES

I found this post but it does not discuss SPARK_MASTER_MEMORY Spark Configuration: SPARK_MEM vs. SPARK_WORKER_MEMORY

Upvotes: 9

Views: 9751

Answers (1)

1esha
1esha

Reputation: 1722

First of all just few words about terms. Spark master is application that coordinates resources allocation from slaves. Master does not perform any computations. Master is just resource manager.

Spark worker is application on worker node which coordinates resources on given worker node.

Spark executor is application created by spark worker which performs tasks on worker node for driver.

Check this doc for additional details - http://spark.apache.org/docs/latest/cluster-overview.html

spark.executor.memory - is amount of memory for executor. This memory used for given user task.

SPARK_WORKER_MEMORY - how much system memory can be used by worker to creating executors on node. For example you have 64gb on node. You set SPARK_WORKER_MEMORY to 60gb. This means that you can create 2 x 30g executors or 10 x 6gb executors and so on.

SPARK_MEM AFAIK is not used anymore. I can not find it in current docs

SPARK_MASTER_MEMORY is memory for master. Should not be to high :)

SPARK_WORKER_CORES is total number of cores to be used by executors by each worker

SPARK_WORKER_INSTANCES is number of workers per worker node.

All these parameters are described here - http://spark.apache.org/docs/latest/spark-standalone.html

Upvotes: 16

Related Questions