CODEWITHSUNDEEP
hadoopmapreducehadoop-yarn
xeon123
xeon123

Reputation: 909

YARN MapReduce has not enough RAM

I am trying to launch my application in Yarn MapReduce. I have set MapReduce in 4 hosts (1 resource manager and 4 node managers). Each host has 2 cores and 4GB of RAM.

When I run my application, it aborts because of lack of RAM [1]. How should I set Yarn MapReduce so that jobs won't run out of RAM?

[1] Error that I have.

2016-02-09 16:01:41,607 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 289.6 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:44,612 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 289.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:47,616 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 289.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:50,621 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 289.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:53,626 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 291.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:56,631 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 295.1 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:59,643 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 295.3 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
 (...)
2016-02-09 16:02:01,173 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001/container_1455032058147_0001_01_000001/syslog
2016-02-09 16:02:01,174 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001/container_1455032058147_0001_01_000001/stdout
2016-02-09 16:02:01,174 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001/container_1455032058147_0001_01_000001/stderr
2016-02-09 16:02:01,250 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001
2016-02-09 16:02:02,644 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1455032058147_0001_01_000001
2016-02-09 16:03:10,146 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-02-09 16:03:11,147 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-02-09 16:03:12,148 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-02-09 16:03:13,149 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

Upvotes: 1

Views: 823

Answers (1)

bachr
bachr

Reputation: 6006

You need to configure the amount of memory given to YARN and it's containers in yarn-site.xml:

<property>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>4096</value>
</property>
<property>
  <name>yarn.scheduler.minimum-allocation-mb</name>
  <value>1024</value>
</property>

Check this sample yarn-site.xml file. As you're running MapReduce jobs you've also to set similar properties in the mapred-site.xmlfile. This article may help you - link.

Upvotes: 3

Related Questions