MapReduce job hangs, waiting for AM container to be allocated

Question

I tried to run simple word count as MapReduce job. Everything works fine when run locally (all work done on Name Node). But, when I try to run it on a cluster using YARN (adding mapreduce.framework.name=yarn to mapred-site.conf) job hangs.

I came across a similar problem here: MapReduce jobs get stuck in Accepted state

Output from job:

*** START ***
15/12/25 17:52:50 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
15/12/25 17:52:51 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
15/12/25 17:52:51 INFO input.FileInputFormat: Total input paths to process : 5
15/12/25 17:52:52 INFO mapreduce.JobSubmitter: number of splits:5
15/12/25 17:52:52 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1451083949804_0001
15/12/25 17:52:53 INFO impl.YarnClientImpl: Submitted application application_1451083949804_0001
15/12/25 17:52:53 INFO mapreduce.Job: The url to track the job: http://hadoop-droplet:8088/proxy/application_1451083949804_0001/
15/12/25 17:52:53 INFO mapreduce.Job: Running job: job_1451083949804_0001

mapred-site.xml:

<configuration>

<property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
</property>

<property>
   <name>mapreduce.job.tracker</name>
   <value>localhost:54311</value>
</property> 

<!--
<property>
   <name>mapreduce.job.tracker.reserved.physicalmemory.mb</name>
   <value></value>
</property>

<property>
   <name>mapreduce.map.memory.mb</name>
   <value>1024</value>
</property>

<property>
   <name>mapreduce.reduce.memory.mb</name>
   <value>2048</value>
</property>    

<property>
   <name>yarn.app.mapreduce.am.resource.mb</name>
   <value>3000</value>
   <source>mapred-site.xml</source>
</property> -->

</configuration>

yarn-site.xml

<configuration>
 <property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
 </property>
 <property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
 </property>

<!--
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>3000</value>
<source>yarn-site.xml</source>
</property>

<property>
  <name>yarn.scheduler.minimum-allocation-mb</name>
  <value>500</value>
</property>

<property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>3000</value>
</property>
-->

</configuration>

//I the left commented options - they were not solving the problem

YarnApplicationState: ACCEPTED: waiting for AM container to be allocated, launched and register with RM.

What can be the problem?

EDIT:

I tried this configuration (commented) on machines: NameNode(8GB RAM) + 2x DataNode (4GB RAM). I get the same effect: Job hangs on ACCEPTED state.

EDIT2: changed configuration (thanks @Manjunath Ballur) to:

yarn-site.xml:

<configuration>
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hadoop-droplet</value>
  </property>

  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>hadoop-droplet:8031</value>
  </property>
  <property>
    <name>yarn.resourcemanager.address</name>
    <value>hadoop-droplet:8032</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>hadoop-droplet:8030</value>
  </property>
  <property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>hadoop-droplet:8033</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>hadoop-droplet:8088</value>
  </property>
  <property>
    <description>Classpath for typical applications.</description>
    <name>yarn.application.classpath</name>
    <value>
        $HADOOP_CONF_DIR,
        $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
        $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
        $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
        $YARN_HOME/*,$YARN_HOME/lib/*
    </value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce.shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
  <property>
    <name>yarn.nodemanager.local-dirs</name>
    <value>/data/1/yarn/local,/data/2/yarn/local,/data/3/yarn/local</value>
  </property>
  <property>
    <name>yarn.nodemanager.log-dirs</name>
    <value>/data/1/yarn/logs,/data/2/yarn/logs,/data/3/yarn/logs</value>
  </property>
  <property>
    <description>Where to aggregate logs</description>
    <name>yarn.nodemanager.remote-app-log-dir</name>
    <value>/var/log/hadoop-yarn/apps</value>
  </property>
  <property> 
    <name>yarn.scheduler.minimum-allocation-mb</name> 
    <value>50</value>
  </property>
  <property> 
    <name>yarn.scheduler.maximum-allocation-mb</name> 
    <value>390</value>
  </property>
  <property> 
    <name>yarn.nodemanager.resource.memory-mb</name> 
    <value>390</value>
  </property>
</configuration>

mapred-site.xml:

<configuration>
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

<property>  
    <name>yarn.app.mapreduce.am.resource.mb</name>  
    <value>50</value>
</property>
<property> 
    <name>yarn.app.mapreduce.am.command-opts</name> 
    <value>-Xmx40m</value>
</property>
<property>
    <name>mapreduce.map.memory.mb</name>
    <value>50</value>
</property>
<property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>50</value>
</property>
<property>
    <name>mapreduce.map.java.opts</name>
    <value>-Xmx40m</value>
</property>
<property>
    <name>mapreduce.reduce.java.opts</name>
    <value>-Xmx40m</value>
</property>
</configuration>

Still not working. Additional info: I can see no nodes on cluster preview (similar problem here: Slave nodes not in Yarn ResourceManager )

Answer 1

Old question, but I got on the same issue recently and in my case it was due to manually setting the master to local in the code.

Please, search for conf.setMaster("local[*]") and remove it.

Hope it helps.

Answer 2

These lines

<property>
  <name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</name>
  <value>100</value>
</property>

in the yarn-site.xml solved my problem since the node will be marked as unhealthy when disk usage is >=95%. Solution mainly suitable for pseudodistributed mode.

Answer 3

The first thing is to check yarn resource manager logs. I had searched the Internet about this problem for a very long time, but nobody told me how to find out what is really happening. It's so straightforward and simple to check yarn resource manager logs. I am confused why people ignore logs.

For me, there was a error in log

Caused by: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=172.16.0.167/172.16.0.167:55622]

That's because I switched wifi network in my work place, so my computer IP changed.

Answer 4

anyway that's work for me .thank you a lot! @KaP

that's my yarn-site.xml

<property>
            <name>yarn.resourcemanager.hostname</name>
            <value>MacdeMacBook-Pro.local</value>
    </property>
    <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
    </property>
    <property>
            <name>yarn.resourcemanager.webapp.address</name>
            <value>${yarn.resourcemanager.hostname}:8088</value>
    </property>
    <property>
       <name>yarn.nodemanager.resource.memory-mb</name>
<value>4096</value>
</property>
<property>
   <name>yarn.scheduler.minimum-allocation-mb</name>
   <value>2048</value>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>2.1</value>

that's my mapred-site.xml

<configuration>
<property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
    </property>

Answer 5

Check your hosts file on master and slave nodes. I had exactly this problem. My hosts file looked like this on master node for example

127.0.0.0 localhost
 127.0.1.1 master-virtualbox
 192.168.15.101 master 

I changed it like below

192.168.15.101 master master-virtualbox localhost

So it worked.

Answer 6

You should check the status of Node managers in your cluster. If the NM nodes are short on disk space then RM will mark them "unhealthy" and those NMs can't allocate new containers.

1) Check the Unhealthy nodes: http://<active_RM>:8088/cluster/nodes/unhealthy

If the "health report" tab says "local-dirs are bad" then it means you need to cleanup some disk space from these nodes.

2) Check the DFS dfs.data.dir property in hdfs-site.xml. It points the location on local file system where hdfs data is stored.

3) Login to those machines and use df -h & hadoop fs - du -h commands to measure the space occupied.

4) Verify hadoop trash and delete it if it's blocking you. hadoop fs -du -h /user/user_name/.Trash and hadoop fs -rm -r /user/user_name/.Trash/*

Answer 7

This has solved my case for this error:

<property>
  <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
  <value>100</value>
</property>

Answer 8

I feel, you are getting your memory settings wrong.

To understand the tuning of YARN configuration, I found this to be a very good source: http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cdh_ig_yarn_tuning.html

I followed the instructions given in this blog and was able to get my jobs running. You should alter your settings proportional to the physical memory you have on your nodes.

Key things to remember is:

• Values of mapreduce.map.memory.mb and mapreduce.reduce.memory.mb should be at least yarn.scheduler.minimum-allocation-mb
• Values of mapreduce.map.java.opts and mapreduce.reduce.java.opts should be around "0.8 times the value of" corresponding mapreduce.map.memory.mb and mapreduce.reduce.memory.mb configurations. (In my case it is 983 MB ~ (0.8 * 1228 MB))
• Similarly, value of yarn.app.mapreduce.am.command-opts should be "0.8 times the value of" yarn.app.mapreduce.am.resource.mb

Following are the settings I use and they work perfectly for me:

yarn-site.xml:

<property> 
    <name>yarn.scheduler.minimum-allocation-mb</name> 
    <value>1228</value>
</property>
<property> 
    <name>yarn.scheduler.maximum-allocation-mb</name> 
    <value>9830</value>
</property>
<property> 
    <name>yarn.nodemanager.resource.memory-mb</name> 
    <value>9830</value>
</property>

mapred-site.xml

<property>  
    <name>yarn.app.mapreduce.am.resource.mb</name>  
    <value>1228</value>
</property>
<property> 
    <name>yarn.app.mapreduce.am.command-opts</name> 
    <value>-Xmx983m</value>
</property>
<property>
    <name>mapreduce.map.memory.mb</name>
    <value>1228</value>
</property>
<property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>1228</value>
</property>
<property>
    <name>mapreduce.map.java.opts</name>
    <value>-Xmx983m</value>
</property>
<property>
    <name>mapreduce.reduce.java.opts</name>
    <value>-Xmx983m</value>
</property>

You can also refer to the answer here: Yarn container understanding and tuning

You can add vCore settings, if you want your container allocation to take into account CPU also. But, for this to work, you need to use CapacityScheduler with DominantResourceCalculator. See the discussion about this here: How are containers created based on vcores and memory in MapReduce2?

Answer 9

You have 512 MB RAM on each of the instance and all your memory configurations in yarn-site.xml and mapred-site.xml are 500 MB to 3 GB. You will not be able to run any thing on the cluster. Change every thing to ~256 MB.

Also your mapred-site.xml is using framework to by yarn and you have job tracker address which is not correct. You need to have resource manager related parameters in yarn-site.xml on a multinode cluster (including resourcemanager web address). With out that, the cluster does not know where your cluster is.

You need to revisit both your xml files.