CODEWITHSUNDEEP
apache-sparkcontainershadoop-yarnworkmanagertaskexecutor
Dajiang Lei
Dajiang Lei

Reputation: 23

Each application submitted by client can launch how many YARN container in each Node Manager?

A container is an abstract notion in YARN. When running Spark on YARN, each Spark executor runs as a YARN container. How many YARN containers can be launched in each Node Manager, by each client-submitted application?

Upvotes: 2

Views: 1058

Answers (2)

Dorjee
Dorjee

Reputation: 1623

Each executor runs in a YARN container.

Depending on how big your YARN cluster is, how your data is spread out among the worker nodes to have better data locality, how many executors you requested for your application, how much resource(cores per executor, memory per executor) you requested per executor and whether your have enabled dynamic resource allocation, Spark decides on how many executors are needed in total and, how many executors to launch per worker nodes.

If you requested for resource that YARN cluster could not accommodate, your requested will be rejected.

Following are the properties to look out for when making spark-submit request.

  • --num-executors - number of total executors you need
  • --executor-cores - number of cores per executor. Max 5 is recommended.
  • --executor-memory - amount of memory per executor.
  • --spark.dynamicAllocation.enabled
  • -- spark.dynamicAllocation.maxExecutors

Upvotes: 0

Joe Widen
Joe Widen

Reputation: 2448

You can run as many executors on a single NodeManager as you want, so long as you have the resources. If you have a server with 20gb RAM and 10 cores, you can run 10 2gb 1core executors on that nodemanager. It wouldn't be advisable to run multiple executors on the same nodemanager as there is overhead cost in shuffling data between executors, even if they process is running on the same machine.

Upvotes: 1

Related Questions