CODEWITHSUNDEEP
hadoophadoop-yarn
kee
kee

Reputation: 11629

How to increase the number of containers in nodemanager in YARN

A node in my YARN cluster has 64GB memory and 24 cores. I set the following properties in the yarn-site.xml:

<property>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>32768</value>
</property>
<property>
  <name>yarn.nodemanager.resource.cpu-vcores</name>
  <value>16</value>
</property>

But I found still the nodemanager in the node has only 7 containers. What are other properties I need to set?

Upvotes: 9

Views: 18490

Answers (2)

avner
avner

Reputation: 785

try something like:

  <property>
    <name>mapreduce.map.memory.mb</name>
    <value>1024</value>      
  </property>

  <property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>2048</value>      
  </property>

Upvotes: 0

Arnon Rotem-Gal-Oz
Arnon Rotem-Gal-Oz

Reputation: 25939

You need to tell YARN how to break down the memory to containers so for instance if you set the memory per container to 2GB will give you 16 containers

<name>yarn.scheduler.minimum-allocation-mb</name>
<value>2048</value>

Upvotes: 17

Related Questions