CODEWITHSUNDEEP
hadoopinstallation
Saras Arya
Saras Arya

Reputation: 3112

hadoop's datanode is not starting

I am using ubuntu 14.04 LTS Java version 8 and Hadoop 2.5.1 for installation. I followed this guide to install all the components. Sorry for not using michael noll's. Now the problem that i face is when i do start-dfs.sh i get the following message 

oroborus@Saras-Dell-System-XPS-L502X:~$ start-dfs.sh <br>
14/11/12 16:12:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable<br>
Starting namenodes on [localhost]<br>
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-oroborus-namenode-Saras-Dell-System-XPS-L502X.out<br>
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-oroborus-datanode-Saras-Dell-System-XPS-L502X.out<br>
Starting secondary namenodes [0.0.0.0]<br>
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-oroborus-secondarynamenode-Saras-Dell-System-XPS-L502X.out<br>
14/11/12 16:12:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable<br>

Now after running start-yarn.sh(which seems to work fine) and jps i get the following output

oroborus@Saras-Dell-System-XPS-L502X:~$ jps
9090 NodeManager
5107 JobHistoryServer
8952 ResourceManager
12442 Jps
11981 NameNode

The ideal output should have datanode in it, but it is not there. Googling and SOing a bit i found out the error is looged in the logs so here are the logs for datanode.(Only error part if you need more let me know)

    2014-11-08 23:30:32,709 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT]
2014-11-08 23:30:33,132 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid dfs.datanode.data.dir /usr/local/hadoop_store/hdfs/datanode :
EPERM: Operation not permitted
        at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
        at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:226)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:642)
        at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:472)
        at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:126)
        at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:142)
        at org.apache.hadoop.hdfs.server.datanode.DataNode$DataNodeDiskChecker.checkDir(DataNode.java:1866)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:1908)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1890)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1782)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1829)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2005)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2029)
2014-11-08 23:30:33,134 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.io.IOException: All directories in dfs.datanode.data.dir are invalid: "/usr/local/hadoop_store/hdfs/datanode/"
        at org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:1917)

Now my doubt is how to make it valid.

Help is appreciated.
P.S. I tried a lot of forums, SO post none of them could give me solution to this problem. Hence the question.

Upvotes: 7

Views: 17413

Answers (4)

Dieter
Dieter

Reputation: 26

When using cloudera, I had the same problem.

* chown hdfs:hadoop all_hdfs_directories
  chown 997:996 all_hdfs_directories (if users don't exist yet)
* chmod 0755 is enough
* delete contents of directories if problem persists

Upvotes: 0

Suraj Ghimire
Suraj Ghimire

Reputation: 41

I also got the same problem,and I fixed them by changing the owner of those working directory.Although you have permissions 777 to these two directory,framework will not be able to use it unless you change the owner to hduser.

$ sudo chown -R hduser:hadoop /usr/local/hadoop/yarn_data/hdfs/namenode

$ sudo chown -R hduser:hadoop /usr/local/hadoop/yarn_data/hdfs/datanode

After this,you again start your cluster and you should see datanode running.

Upvotes: 4

The joker
The joker

Reputation: 189

  1. first stop all the entities like namenode, datanode etc. (you will be having some script or command to do that)
  2. Format tmp directory
  3. Go to /var/cache/hadoop-hdfs/hdfs/dfs/ and delete all the contents in the directory manually
  4. Now format your namenode again
  5. start all the entities then use jps command to confirm that the datanode has been started
  6. Now run whichever application you may like or have.

Hope this helps.

Upvotes: 1

Ramanan
Ramanan

Reputation: 1000

First delete all contents from hdfs folder:

Value of <name>hadoop.tmp.dir</name>

rm -rf /usr/local/hadoop_store

Make sure that dir has right owner and permission /usr/local/hadoop_store

hduser@localhost$sudo chown hduser:hadoop -R /usr/local/hadoop_store
hduser@localhost$sudo chmod 777 -R /usr/local/hadoop_store

Format the namenode:

hduser@localhost$hadoop namenode -format

Start all processes again

Upvotes: 12

Related Questions