CODEWITHSUNDEEP
hadoopmapreduce
Trams
Trams

Reputation: 421

Determin whether slave nodes in hadoop cluster has been assigned tasks

I'm new to Hadoop and MapReduce. I just deployed a Hadoop cluster with one master machine and 32 slave machines. However when I start to run an example program, it seems that it just runs to slow. How can I determine whether a map/reduce task has really been assigned to a slave node for execution?

The example program is executed like that:

hadoop jar ${HADOOP_HOME}/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar pi 32 100

Upvotes: 1

Views: 987

Answers (2)

maxteneff
maxteneff

Reputation: 1531

The easiest way to take a look at Yarn Web UI. By default it uses port 8088 on your master node (change master in the URI by your own IP address):

http://master:8088/cluster

There you can see total resources of your cluster and list of all applications. For every application you can find out how many mappers/reducers were used and where (on what machine) they were executed.

Upvotes: 0

Sazzad
Sazzad

Reputation: 853

okay lots of possibilities there. Hadoop comes out to help in distributed task.

  1. So if your code is written in way that everything is dependent then there is no use of 32 slaves. rather it will take overhead time to manage connection.
  2. check your hadoopMasterIp:50070 if if all the datanodes(slave) is running or not. obviously if you did not change dfs.http.address in your core-site.xml.

Upvotes: 1

Related Questions