CODEWITHSUNDEEP
hadoopmapreducehdfs
Sachin
Sachin

Reputation: 18777

Datanode capacity is 0kb

I am trying to get my hands dirty with Hadoop. My question might be pretty basic, please bear with me.

I am reading Hadoop: A Definitive Guide and following the weather data tutorial. While copying data to HDFS, I am getting following error:

13/09/02 16:34:35 ERROR hdfs.DFSClient: Failed to close file /user/bhushan/gz/home/bhushan/ncdc_data/ftp3.ncdc.noaa.gov/pub/data/noaa/1901.gz
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/bhushan/gz/home/bhushan/ncdc_data/ftp3.ncdc.noaa.gov/pub/data/noaa/1901.gz could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783)
    at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)

Something is definitely not right with my setup. When I see the report, this is what I get:

bhushan@ubuntu:~/Documents/hadoop-1.2.1/bin$ hadoop dfsadmin -report
Configured Capacity: 0 (0 KB)
Present Capacity: 0 (0 KB)
DFS Remaining: 0 (0 KB)
DFS Used: 0 (0 KB)
DFS Used%: �%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 0 (0 total, 0 dead)

The 3 config files are as follows (all as per the book):

hdfs-site.xml:

<?xml version="1.0"?>
<!-- hdfs-site.xml -->
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>

core-site.xml:

<?xml version="1.0"?>
<!-- core-site.xml -->
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost/</value>
</property>
</configuration>

mapred-site.xml:

<?xml version="1.0"?>
<!-- mapred-site.xml -->
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:8021</value>
</property>
</configuration>

I formatted HDFS several times, but it did not help.

Do I need to specify the HDFS size somewhere explicitly? From the book:

Datanodes are not involved in the initial formatting process, since the namenode manages all of the filesystem’s metadata, and datanodes can join or leave the cluster dynamically. For the same reason, you don’t need to say how large a filesystem to create, since this is determined by the number of datanodes in the cluster, which can be increased as needed, long after the filesystem was formatted.

Upvotes: 2

Views: 4211

Answers (1)

Rags
Rags

Reputation: 1881

I think your DataNode process is not running. I guess you are working on a Pseudo cluster. run "jps" command and ensure DataNode process is running and sustains for some time like 4 to 5 minutes. If DataNode is running or it dies down in few minutes there is some problem in configurations. You can try the following solution.

Stop the cluster. Remove the DataNode persistence directory. You must have configured it using "dfs.data.dir" property in hdfs-site.xml. If you have not configured then it will use the Linux users temporary directory. Locate that directory and remove. Then start the cluster again. Try copying the file again and it should work.

Upvotes: 3

Related Questions