CODEWITHSUNDEEP
apache-spark
Knows Not Much
Knows Not Much

Reputation: 31526

Default number of executors and cores for spark-shell

If I run a spark program in spark shell, is it possible that the program can hog the entire hadoop cluster for hours?

usually there is a setting called num-executors and executor-cores.

spark-shell --driver-memory 10G --executor-memory 15G --executor-cores 8

but if they are not specified and I just run "spark-shell"... will it consume the entire cluster? or are there reasonable defaults.

Upvotes: 11

Views: 28862

Answers (1)

Daniel de Paula
Daniel de Paula

Reputation: 17862

The default values for most configuration properties can be found in the Spark Configuration documentation. For the configuration properties on your example, the defaults are:

  • spark.driver.memory = 1g
  • spark.executor.memory = 1g
  • spark.executor.cores = 1 in YARN mode, all the available cores on the worker in standalone mode.

Additionally, you can override these defaults by creating the file$SPARK-HOME/conf/spark-defaults.conf with the properties you want (as described here). Then, if the file exists with the desired values, you don't need to pass them as arguments to the spark-shell command.

Upvotes: 15

Related Questions