CODEWITHSUNDEEP
apache-sparkhadoophadoop-yarnvirtual-memory
Abbas Gadhia
Abbas Gadhia

Reputation: 15118

Apache Yarn - Allocating more memory than the Physical memory or RAM

I was considering changing yarn.nodemanager.resource.memory-mb to a value higher than the RAM available on my machine. Doing a quick search revealed that not many people are doing this.

Many long lived applications on yarn, are bound to have a jvm heap space allocation in which some of their memory is more frequently used and some of it is rarely used. In this case, it would make perfect sense for such applications to have some of their infrequently used memory portions swapped to disk and reallocating the available physical memory to other applications that need it.

Given the above background, can someone either please corroborate my reasoning or offer an alternate perspective? Also, can you please also clarify how the parameter yarn.nodemanager.vmem-pmem-ratio would work in the above case?

Upvotes: 1

Views: 951

Answers (1)

tk421
tk421

Reputation: 5967

This is not a good idea. Trying to use more memory than what is available will eventually crash your Node Manager hosts.

There already is a feature called opportunistic containers which uses spare memory not used by the NMs and adds more containers to those hosts. Refer to:

YARN-1011 [Umbrella] Schedule containers based on utilization of currently allocated containers

In addition, Pepperdata has a product that does almost the same thing if you can't wait for YARN-1011.

https://www.pepperdata.com/products/capacity-optimizer/

As for yarn.nodemanager.vmem-pmem-ratio, don't enable this as it's not recommended anymore.

YARN-782 vcores-pcores ratio functions differently from vmem-pmem ratio in misleading way

Upvotes: 1

Related Questions