CODEWITHSUNDEEP
hadoopmapreducehortonworks-data-platformapache-tez
Aravind Yarram
Aravind Yarram

Reputation: 80194

What does VIRTUAL_MEMORY_BYTES task counter mean in Hadoop?

The following excerpt from The Definitive Guide provides high level details as shown below but

  1. what exactly is virtual memory is referring to in this task counter?
  2. How to interpret it? How is it related to PHYSICAL_MEMORY_BYTES?

enter image description here

Following is an example extract from one of the jobs. Physical is 214 GB approx. and virtual is 611 GB approx.

enter image description here

Upvotes: 0

Views: 863

Answers (1)

Taha Naqvi
Taha Naqvi

Reputation: 1766

1.What exactly is virtual memory is referring to in this task counter?

 Virtual Memory here is used to prevent Out of Memory errors of a task,if data size doesn't fits in RAM(physical mem).
  in RAM.So a portion of memory of size what didn't fit in RAM will be used as Virtual Memory.

So,while setting up hadoop cluster one is advised to have the value of vm.swappiness =1 to achieve better performance. On linux systems, vm.swappiness is set to 60 by default. Higher the value more aggresive swapping of memory pages.

https://community.hortonworks.com/articles/33522/swappiness-setting-recommendation.html

2. How to interpret it? How is it related to PHYSICAL_MEMORY_BYTES?

swapping of memory pages from physical memory to virtual memory on disk when not enough phy mem

This is the relation between PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES.

Upvotes: 2

Related Questions