CODEWITHSUNDEEP
apache-sparkhadoop-yarnnodemanagercluster-mode
Saiteja Parsi
Saiteja Parsi

Reputation: 454

Can two executors / drivers from different Spark applications run on same node in cluster mode?

I read an article in Medium which claims that the number of executors + 1 (for driver), should be a multiple of 3, to efficiently utilize the core on a machine (16 cores, in this case, i.e, 5 per executor and 1 will be reserved for OS and node manager)

I am unable to validate this statement using experimenting on the cluster due to practical reasons. Did anybody try this? or have reference to code/documentation stating Yarn nodes will/not share cluster resources between another Spark application?

Upvotes: 0

Views: 1276

Answers (1)

Ged
Ged

Reputation: 18108

It's a big question, but in short - basing on the title and YARN in the text:

  • You get resources allocated by YARN that you requested via Spark(submit).

  • A Node has many Executors.

  • You cannot share an Executor at the same time, but the Executor can be relinquished if YARN Dynamic Resource Allocation is in effect, after a Stage has completed.

  • As a Node has many Executors, many Spark Apps can run their Tasks concurrently on the same Node, Worker, if they were granted those resources.

Upvotes: 1

Related Questions