CODEWITHSUNDEEP
apachehadoopamazon-web-services
lucTiber
lucTiber

Reputation: 431

Hadoop 272 - Datanodes starts and then stop

The Env details:

I installed a hadoop 2.7.2 (not HW but pure Hadoop) multinode cluster on AWS (1 Namenode/1 2nd NN/ 3 datanodes - ubuntu 14.04).

The cluster was based on the following tutorial (http://mfaizmzaki.com/2015/12/17/how-to-install-hadoop-2-7-1-multi-node-cluster-on-amazon-aws-ec2-instance-improved-part-1/) --> this means the first install (the master) is copied across and tuned

The Issue:

The 3 data nodes individually work correctly if I configure the cluster with 1 Datanode (I specifically excluded the 2 others).

As soon as I add another data node the data node booting first log a FATAL error (see extract of the log file hereafter and snapshot of the VERSION file) and stop. The data node booting second work then fine...

Any idea-recommendation ? Am I doing something wrong cloning the AMI of the master on other machine? Thanks Folks!

Log File

INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Unsuccessfully sent block report 0x1858458671b, containing 1 storage report(s), of which we sent 0. The reports had 0 total blocks and used 0 RPC(s). This took 5 msec to generate and 35 msecs for RPC and NN processing. Got back no commands.

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1251070591-172.Y.Y.Y-1454167071207 (Datanode Uuid 54bc8b80-b84f-4893-8b96-36568acc5d4b) service to master/172.Y.Y.Y:9000 is shutting down org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.UnregisteredNodeException): Data node DatanodeRegistration(172.X.X.X:50010, datanodeUuid=54bc8b80-b84f-4893-8b96-36568acc5d4b, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-56;cid=CID-8e09ff25-80fb-4834-878b-f23b3deb62d0;nsid=278157295;c=0) is attempting to report storage ID 54bc8b80-b84f-4893-8b96-36568acc5d4b. Node 172.Z.Z.Z:50010 is expected to serve this storage.

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool BP-1251070591-172.31.34.94-1454167071207 (Datanode Uuid 54bc8b80-b84f-4893-8b96-36568acc5d4b) service to master/172.Y.Y.Y:9000

INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool BP-1251070591-172.Y.Y.Y-1454167071207 (Datanode Uuid 54bc8b80-b84f-4893-8b96-36568acc5d4b) 

INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Removing block pool BP-1251070591-172.31.34.94-1454167071207

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0

INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:    /************************************************************ SHUTDOWN_MSG: Shutting down DataNode at HNDATA2/172.X.X.x ************************************************************/

Upvotes: 0

Views: 365

Answers (1)

Thanga
Thanga

Reputation: 8161

You have to add IP addresses of the three datanodes in slaves file of the namenode. then restart the cluster. This will fix the issue

slaves

<IPaddress of datanode1>
<IPaddress of datanode2>
<IPaddress of datanode3>

Upvotes: 0

Related Questions