CODEWITHSUNDEEP
hadoopamazon-ec2
Aman Chawla
Aman Chawla

Reputation: 724

DataNode can't talk to NameNode in Hadoop 2.2

I'm setting up a hadoop 2.2 cluster. I have successfully configured a master and a slave. When I enter start-dfs.sh and start-yarn.sh on the master, all the daemons start correctly.

To be specific, on the master the following are running:

On the slave, the following are running:

When I open http://master-host:50070 I see that there is only 1 "Live Node" and it is referring to the datanode on the master.

The datanode on the slave is started, but not being able to tell the master that it started. This is the only error I can find:

From /logs/hadoop-hduser-datanode.log on the slave: WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: ec2-xx-xxx-xx-xx.compute-1.amazonaws.com/xx.xxx.xx.xxx:9001

Things I have checked/verified:

Any suggestions are greatly appreciated.

Upvotes: 1

Views: 3055

Answers (1)

Aman Chawla
Aman Chawla

Reputation: 724

My issue was in the hosts file:

The hosts file on the slave and master needed to be (they're identical_:

127.0.0.1 localhost
<master internal ip> master
<slave internal ip> slave

For AWS you need to use the internal ip that is something like xx.xxx.xxx.xxx (not the external ip in the ec2-xx-xx-xxx.aws.com and not the ip-xx-xx-xxx).

Also, core-site.xml should refer to the location of hdfs as http://master:9000.

Upvotes: 1

Related Questions