How does spark dynamic resource allocation work on YARN (with regards to NodeManagers)?

Question

Let's assume that I have 4 NM and I have configured spark in yarn-client mode. Then, I set dynamic allocation to true to automatically add or remove a executor based on workload. If I understand correctly, each Spark executor runs as a Yarn container.

So, if I add more NM will the number of executors increase ?

If I remove a NM while a Spark application is running, something will happen to that application?

Can I add/remove executors based on other metrics ? If the answer is yes, there is a function, preferably in python,that does that ?

Answer 1

If I understand correctly, each Spark executor runs as a Yarn container.

Yes. That's how it happens for any application deployed to YARN, Spark including. Spark is not in any way special to YARN.

So, if I add more NM will the number of executors increase ?

No. There's no relationship between the number of YARN NodeManagers and Spark's executors.

From Dynamic Resource Allocation:

Spark provides a mechanism to dynamically adjust the resources your application occupies based on the workload. This means that your application may give resources back to the cluster if they are no longer used and request them again later when there is demand.

As you may have guessed correctly by now, it is irrelevant how many NMs you have in your cluster and it's by the workload when Spark decides whether to request new executors or remove some.

If I remove a NM while a Spark application is running, something will happen to that application?

Yes, but only when Spark uses that NM for executors. After all, NodeManager gives resources (CPU and memory) to a YARN cluster manager that will in turn give them to applications like Spark applications. If you take them back, say by shutting the node down, the resource won't be available anymore and the process of a Spark executor simply dies (as any other process with no resources to run).

Can I add/remove executors based on other metrics ?

Yes, but usually it's Spark job (no pun intended) to do the calculation and requesting new executors.

You can use SparkContext to manage executors using killExecutors, requestExecutors and requestTotalExecutors methods.

killExecutor(executorId: String): Boolean Request that the cluster manager kill the specified executor.

requestExecutors(numAdditionalExecutors: Int): Boolean Request an additional number of executors from the cluster manager.

requestTotalExecutors(numExecutors: Int, localityAwareTasks: Int, hostToLocalTaskCount: Map[String, Int]): Boolean Update the cluster manager on our scheduling needs.