How is memory for Spark on EMR calculated/provisioned?

Question

I am running Spark jobs on EMR with YARN and don't understand the provisioning and reporting of memory from the UIs. I have a master and one core node instance r4.8xlarge which should have 32 cores and 244 GB of memory. According to this doc, it should have 241 GB allocated to YARN. Looking at the UI, this number is 236 GB probably due to additional overheads. Based on best practices, I have configured job to have below configurations.

--executor-cores 5 --executor-memory 35GB --num-executors 6 --conf spark.dynamicAllocation.enabled=false

Calculation for executor memory (236 GB / 6 executors) * 0.9 = 35 GB

When I submit a spark job and I look at Spark UI or console for executor metrics, the numbers are very different and I am confused as to how these are calculated and provisioned. Instead of 6 executors, there are only 4 which results in the job only using 20 cores instead of the available 30. The amount of memory for each executor is 22.2 GB instead of 35 GB which is only 88 GB out of the total 236 GB available.

I have looked at many resources but they only talk about how to tune spark jobs by setting YARN and Spark config which I have followed yet the results are unexpected.

Can someone help explain?

edit: The only applications installed on the cluster are Spark and Hadoop.

Answer 1

Memory

This is due to the spark memory management.

Quoting from From: https://www.tutorialdocs.com/article/spark-memory-management.html

By default, Spark uses On-heap memory only. The size of the On-heap memory is configured by the –executor-memory or spark.executor.memory parameter when the Spark Application starts. The concurrent tasks running inside Executor share JVM's On-heap memory.

The On-heap memory area in the Executor can be roughly divided into the following four blocks:

Storage Memory: It's mainly used to store Spark cache data, such as RDD  
 cache, Broadcast variable, Unroll data, and so on.

Execution Memory: It's mainly used to store temporary data in the calculation 
 . process of Shuffle, Join, Sort, Aggregation, etc.

User Memory: It's mainly used to store the data needed for RDD conversion  
 operations, such as the information for RDD dependency.

Reserved Memory: The memory is reserved for system and is used to store 
Spark's internal objects.

The available memory that you see in the dashboard is the 75% of the allocated memory.

The total allocated memory per executor may vary due to available memory in the node. The exact 236GB might not be available for the yarn. The datanode process etc might take some more memory.

The memory you see is the storage memory. Storage memory + Execution memory = 75% of total memory allocated.

For more information:

1. https://databricks.com/session/deep-dive-apache-spark-memory-management
2. https://0x0fff.com/spark-memory-management/
3. https://www.tutorialdocs.com/article/spark-memory-management.html

Executor count

You need to check yarn.nodemanager.resource.memory-mb in the yarn-site.xml file. It denotes "the total memory that a single NodeManager can allocate across all containers on one node". It might be the case that the yarn hasn't given all the memory available in the box. Therefore, the spark is not able to negotiate the 6 executors.

One more thing spark.yarn.executor.memoryOverhead is set to 384MB per executor unless it is overridden. This needs to be added into the calculation.

For more information

1. Apache Spark: setting executor instances does not change the executors

How UI Calculates the memory

1. How does web UI calculate Storage Memory (in Executors tab)?