Difference between executor and container in Spark

Question

I am trying to clearly understand how memory allocation happens in a yarn managed cluster. I understand that there are a bunch of executors (one executor having its own JVM) and one executor can have one or more vcores during execution.

I am trying to tie up this understand in YARN configuration where things are segregated as Containers. Each container is actually a mix of some Vcores and fraction of heap memory.

Can someone confirm if one executor gets one container or one executor can have more than one containers. I read some documentation on Cloudera on YARN memory management and it appears to be saying that Container has an Executor allocated to it.

Cloudera Memory Management

Answer 1

Spark Executor runs within a Yarn Container, not across Containers.

A Yarn Container is provided by the YARN Resource Manager on demand - at start of Spark Application of via YARN Dynamic Resource Allocation.

A Yarn Container can have only one Spark Executor, but 1 or indeed more Cores can be assigned to the Executor.

Each Spark Executor and Driver runs as part of its own YARN Container.

Executors run on a given Worker.

Moreover, everything is in the context of an Application and as such an Application has Executors on many Workers.

Answer 2

When running Spark on YARN, each Spark executor runs as a YARN container.