CODEWITHSUNDEEP
apache-spark
Bamqf
Bamqf

Reputation: 3542

Mapping between Spark configurations and parameters

When I launch spark in command line, I found parameter num-executors acts similar as spark.executor.instances in the configuration file. Are they actually the same? If so, where can I find a full mapping between all such pairs of same functionality?

Upvotes: 0

Views: 51

Answers (1)

KrisP
KrisP

Reputation: 1216

From the documentation

The Spark shell and spark-submit tool support two ways to load configurations dynamically. The first are command line options, such as --master, as shown above. spark-submit can accept any Spark property using the --conf flag, but uses special flags for properties that play a part in launching the Spark application. Running ./bin/spark-submit --help will show the entire list of these options.

So, there are way fewer command line options such as --executor-cores than there is spark options such as spark.executor.cores, and like documentation says, you get those -- options by running ./bin/spark-submit --help. Do run this - it will tell you that not all CL options are usable in every situation (something that confused me a lot). You can set any property that does not have a special command line option like so: --conf spark.executor.cores=16.

Here is an incomplete table - you must read through the comments to find the appropriate parameters.

Upvotes: 1

Related Questions