CODEWITHSUNDEEP
apache-spark
SRN
SRN

Reputation: 503

How many executors will be created in spark in standalone

I have spark installed in a server (8vCores and 64GB RAM) without hadoop/yarn. Am running a job with the below properties.

 spark.executor.instances=4
 spark.executor.memory=6g
 spark.executor.cores=4
 spark.driver.memory=16g

From the top command, I see that the program is taking around 22.3gb memory. Could you please let me know how many executors will be created when we run in standalone.

Thanks

Upvotes: 0

Views: 264

Answers (1)

Ged
Ged

Reputation: 18108

From the docs https://spark.apache.org/docs/latest/spark-standalone.html

The Spark Standalone cluster mode currently only supports a simple FIFO scheduler across applications. You can start such a Cluster in different ways.

However, to allow multiple concurrent users, you can control the maximum number of resources each application will use.

By default, it will acquire all cores in the cluster, which only makes sense if you just run one application at a time. Who would do this? The exception proves the rule of course.

You can cap the number of cores by setting spark.cores.max in your SparkConf. For example:

val conf = new SparkConf()
  .setMaster(...)
  .setAppName(...)
  .set("spark.cores.max", "10")
val sc = new SparkContext(conf)

In addition, you can configure spark.deploy.defaultCores on the cluster master process to change the default for applications that don’t set spark.cores.max to something less than infinite. Do this by adding the following to conf/spark-env.sh:

export SPARK_MASTER_OPTS="-Dspark.deploy.defaultCores=<value>"

This is useful on shared clusters where users might not have configured a maximum number of cores individually.

You need to look at it more from the cores perspective.

Upvotes: 1

Related Questions