CODEWITHSUNDEEP
apache-sparkhadoop-yarn
Raj Nandini Padhy
Raj Nandini Padhy

Reputation: 1

Does executors in spark nd application master in yarn do the same job?

Does executors in spark nd application master in yarn do the same

Upvotes: 0

Views: 79

Answers (1)

Shreyas B
Shreyas B

Reputation: 505

In Spark, there is a Driver and Executors. I'm not gonna go into detail of what driver and executors are but in a one-liner, the driver manages the job flow and schedules tasks, and Executors are worker nodes processes in charge of running individual tasks.

YARN is basically a resource manager which allocates memory to compute engines. Now, this compute engine can be Spark/Tez/Map-reduce. What you need to understand here is when YARN successfully allocates memory they are called containers.

Now when Spark Job is deployed in YARN, Assuming that YARN has sufficient memory for the spark job to run, Yarn first allocates resources as containers for Spark Application Master which will have the driver program (in case of cluster mode). This Application Master will further requests resources for spark executors which YARN will further allocate as containers. So spark job will have multiple containers, one for the driver program and n containers for n executors. So you see in the computing sense the fundamental difference between spark running in a spark cluster and spark running in YARN is the use of containers.

So executors and application master in YARN run inside containers and do the same thing as spark on spark clusters.

Upvotes: 1

Related Questions