CODEWITHSUNDEEP
apache-sparkhadoop-yarn
fonhorst
fonhorst

Reputation: 65

How to set several Apache Spark executors per node in YARN?

I wonder how to set several Apache Spark executors per node in YARN. Do I need specify it somehow in yarn-site.xml?

Upvotes: 0

Views: 2358

Answers (1)

Arnon Rotem-Gal-Oz
Arnon Rotem-Gal-Oz

Reputation: 25909

When spark runs it behaves like any other YARN application so it asks Yarn for resources so for starters you need to set up Yarn so that it would be able to accommodate your executors (Hortonworks has some nice guidelines here).

Then you can set desired number of executors and their memory when you submit a job as in the example that appears in Spark's documentation

$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi \
    --master yarn-cluster \
    --num-executors 3 \
    --driver-memory 4g \
    --executor-memory 2g \
    --executor-cores 1 \
    lib/spark-examples*.jar \

Upvotes: 1

Related Questions