CODEWITHSUNDEEP
scalaapache-sparkpyspark
J. Doe
J. Doe

Reputation: 1

Why do we need more executors than number of machines in Spark?

What's the logic behind requesting more executors than machines available in your cluster?

In the ideal situation, we would like to have 1 executor (=1 jvm) at each of our machines, and not few in each machine.
If not, then why?

Thanks in advance

Upvotes: 0

Views: 114

Answers (1)

user10463510
user10463510

Reputation: 11

In the ideal situation, we would like to have 1 executor (=1 jvm) at each of our machines, and not few in each machine.

Not necessarily. Depending on the amount of available memory and JVM implementation separate virtual machines can be much a better option, in particular to:

  • Improve memory management with large machines - see for example Why 35GB Heap is Less Than 32GB – Java JVM Memory Oddities.
  • To improve fault tolerance with unstable workloads - if one JVM fails you'll lose work for all corresponding threads, so keeping things smaller can keep things under control.
  • To minimize effort required for GC tuning - very large instances can be extremely painful to tune.

Upvotes: 1

Related Questions