CODEWITHSUNDEEP
apache-sparkpysparkspark-webui
kee
kee

Reputation: 11629

Explanation about Executor Summary in Spark Web UI

I have a few questions about the Executor section of Spark Web UI:

  1. I see two numbers such as 0.0 B / 434.4 MiB under Storage Memory, On Heap Storage Memory, and Off Heap Storage Memory, what are those?

  2. Also is Storage Memory the sum of On Heap Storage Memory and Off Heap Storage Memory?

  3. With regard to Off Heap Storage Memory, does it purely come from OffHeap memory (set by spark.memory.offHeap.size) or include spark.executor.memoryOverhead? Not so sure the latter can be used for the dataframe storage though.

Executor tab of Spark Web UI

Upvotes: 4

Views: 411

Answers (1)

Koedlt
Koedlt

Reputation: 5973

I'm not exactly sure of which version you're on, so I'll make this answer for version 3.3.1 (latest version at the time of writing this post):

  1. We can understand what those 2 numbers are by looking at the HTML code that generates this page.

    • Storage Memory: Memory used / total available memory for storage of data like RDD partitions cached in memory.
    • On Heap Storage Memory: Memory used / total available memory for on heap storage of data like RDD partitions cached in memory.
    • Off Heap Storage Memory: Memory used / total available memory for off heap storage of data like RDD partitions cached in memory.

  2. Storage Memory is indeed the sum of On Heap and Off heap memory usage, both for:

  /**
   * Storage memory currently in use, in bytes.
   */
  final def storageMemoryUsed: Long = synchronized {
    onHeapStorageMemoryPool.memoryUsed + offHeapStorageMemoryPool.memoryUsed
  }

  /** Total amount of memory available for storage, in bytes. */
  private def maxMemory: Long = {
    memoryManager.maxOnHeapStorageMemory + memoryManager.maxOffHeapStorageMemory
  }
  1. The off heap storage memory comes purely from the spark.memory.offHeap.size parameter, as can be seen here:
  protected[this] val maxOffHeapMemory = conf.get(MEMORY_OFFHEAP_SIZE)
  protected[this] val offHeapStorageMemory =
    (maxOffHeapMemory * conf.get(MEMORY_STORAGE_FRACTION)).toLong

This MEMORY_OFFHEAP_SIZE is defined by spark.memory.offHeap.size:

  private[spark] val MEMORY_OFFHEAP_SIZE = ConfigBuilder("spark.memory.offHeap.size")
    .doc("The absolute amount of memory which can be used for off-heap allocation, " +
      " in bytes unless otherwise specified. " +
      "This setting has no impact on heap memory usage, so if your executors' total memory " +
      "consumption must fit within some hard limit then be sure to shrink your JVM heap size " +
      "accordingly. This must be set to a positive value when spark.memory.offHeap.enabled=true.")
    .version("1.6.0")
    .bytesConf(ByteUnit.BYTE)
    .checkValue(_ >= 0, "The off-heap memory size must not be negative")
    .createWithDefault(0)

Upvotes: 2

Related Questions