CODEWITHSUNDEEP
apache-spark
samthebest
samthebest

Reputation: 31533

Change Executor Memory (and other configs) for Spark Shell

How to change executor memory (and other configs) for Apache Spark Shell?

In particular I would like to give flats to spark-shell, like -Dspark-cores-max=12 when I start it so that my jobs in the spark shell will use those configuration settings.

Upvotes: 37

Views: 40818

Answers (4)

Marreddy
Marreddy

Reputation: 21

For configuring Cores and Memory for executors.

spark-shell --help

--master MASTER_URL spark://host:port, mesos://host:port, yarn,
--executor-memory MEM Memory per executor (e.g. 1000M, 2G) (Default: 1G).
--total-executor-cores NUM Total cores for all executors.
--executor-cores NUM Number of cores used by each executor. (Default: 1 in YARN and K8S modes, or all available cores on the worker in standalone mode).
--num-executors NUM Number of executors to launch (Default: 2).If dynamic allocation is enabled, the initial number of

Final commands : If your system is having 6 Cores and 6GB RAM.

CASE 1 : creates 6 executors with each 1 core and 1GB RAM

spark-shell --master spark://sparkmaster:7077 --executor-cores 1 --executor-memory 1g

CASE 2 : creates 3 executors with each 1 core and 2GB RAM. The Max memory is 6GB, 3 cores are ideal.

spark-shell --master spark://sparkmaster:7077 --executor-cores 1 --executor-memory 2g

CASE 3 : creates 2 executors with each 3 cores and 3GB RAM. Using all RAM and Cores

spark-shell --master spark://sparkmaster:7077 --executor-cores 3 --executor-memory 3g

CASE 4 : creates 2 executors with each 3 cores and only 1GB RAM.

spark-shell --master spark://sparkmaster:7077 --executor-cores 3 --executor-memory 1g

CASE 5 : if we want to use only one executors with 1 core and 1GB RAM

spark-shell --master spark://sparkmaster:7077 --total-executor-cores 1 --executor-cores 1 --executor-memory 1g

CASE 6 : if we want to use only two executors with each 1 core and 1GB RAM

spark-shell --master spark://sparkmaster:7077 --total-executor-cores 2 --executor-cores 1 --executor-memory 1g

CASE 7 : if we want to use only two executors with each 2 cores and 2GB RAM (total 4 cores and 4GB RAM)

spark-shell --master spark://sparkmaster:7077 --total-executor-cores 4 --executor-cores 2 --executor-memory 2g

CASE 8 : If we apply --total-executor-cores 2, then only one executor will be created.

spark-shell --master spark://sparkmaster:7077 --total-executor-cores 4 --executor-cores 2 --executor-memory 2g

CASE 9 : Total executor cores: 3 is not divisible by cores per executor: 2, the left cores: 1 will not be allocated. one executor with 2 core will create.

spark-shell --master spark://sparkmaster:7077 --total-executor-cores 3 --executor-cores 2 --executor-memory 2g

So --total-executor-cores / --executor-cores = Number of executors that will create.

Upvotes: 1

Orhan Celik
Orhan Celik

Reputation: 1575

If you are running the spark-shell on spark installed on standalone mode (1 node), use

./bin/spark-shell --driver-memory 4g

If you are running the spark-shell on spark installed on cluster (2+ nodes), use

./bin/spark-shell --executor-memory 4g

4g is 4GB.

Upvotes: 3

samthebest
samthebest

Reputation: 31533

DEPRECATED USE ACCEPTED ANSWER

Write a script like this:

#!/bin/bash
export SPARK_JAVA_OPTS="$*"
MASTER=spark://ec2-99-99-99-99:7077 /usr/share/spark/bin/spark-shell

/usr/share/spark/bin/spark-shell should be the path to where the long spark-shell starting script is. On my cluster there was another script in /usr/local/bin/ but this one was just a few lines similar to above and had SPARK_JAVA_OPTS hardcoded.

Anyway, example use:

my-spark-starter-script -Dspark-cores-max=12 -Dspark.executor.memory=26000m

Upvotes: 1

Junaid
Junaid

Reputation: 778

As of spark 1.2.0 you can set memory and cores by giving following arguments to spark-shell.

spark-shell --driver-memory 10G --executor-memory 15G --executor-cores 8

to see other options you can give following commands to spark shell 

spark-shell --help

Upvotes: 57

Related Questions