CODEWITHSUNDEEP
apache-sparkgoogle-cloud-dataproc
Khilesh Chauhan
Khilesh Chauhan

Reputation: 881

How to set --num-executors in dataproc cluster

We have a dataproc cluster with 10 Nodes and unable to understand how to set the parameter for --num-executor for spark jobs.

Below are the points which are confusing -

Upvotes: 2

Views: 209

Answers (1)

Koedlt
Koedlt

Reputation: 6011

An executor is just a Java process running running on their own JVM on some machine somewhere.

To answer your questions:

  • Can we have less executor than number of worker nodes. If yes what will happen to idle worker nodes.
    • Yes you can. That just means that the idle worker nodes won't have this Java process running on there.
  • Can we have more executor than number of worker nodes. where additional executor will run?
    • Yes you can, provided a single worker node can host your 2 JVMs (enough memory/cpu/...) and your resource manager can handle it. The resource manager will decide which node will be hosting the additional executor process.
  • Can we run more than 1 spark application running using 1 executor per node?
    • This means that you'll have more than 1 executor on a single worker node, so same as point 2.
  • Can we run 1 spark application using more than 1 executor per node ?
    • This means that you'll have more than 1 executor on a single worker node, so same as point 2.

Hope this helps!

Upvotes: 1

Related Questions