CODEWITHSUNDEEP
hadoop
stholy
stholy

Reputation: 322

Hadoop tasktracker issue

This is a weird problem. I was running a MR job (single node) when my lappy lost power and shutdown itself. Thanks to the lack of "no battery" warning in ubuntu 10.04

When I started up again and tried to start hadoop (single node), everything goes well and when I issue "jps" from the command line, all the master and slave processes are listed. I also tried issuing "netstat -plten | grep java".

Output from command line:

hduser@fox:~$ jps
10476 DataNode
1819 
10696 JobTracker
10813 TaskTracker
11741 Jps
10605 SecondaryNameNode
10360 NameNode

hduser@fox:~$ sudo netstat -plten | grep java
tcp        0      0 0.0.0.0:55376           0.0.0.0:*               LISTEN      1000       141925      10476/java      
tcp        0      0 0.0.0.0:42352           0.0.0.0:*               LISTEN      1000       141790      10360/java      
tcp        0      0 0.0.0.0:50070           0.0.0.0:*               LISTEN      1000       142727      10360/java      
tcp        0      0 0.0.0.0:50010           0.0.0.0:*               LISTEN      1000       142818      10476/java      
tcp        0      0 0.0.0.0:50075           0.0.0.0:*               LISTEN      1000       143308      10476/java      
tcp        0      0 0.0.0.0:50020           0.0.0.0:*               LISTEN      1000       146358      10476/java      
tcp        0      0 127.0.0.1:38533         0.0.0.0:*               LISTEN      1000       143217      10813/java      
tcp        0      0 102.173.156.27:54310   0.0.0.0:*               LISTEN      1000       141913      10360/java      
tcp        0      0 102.173.156.27:54311   0.0.0.0:*               LISTEN      1000       142359      10696/java      
tcp        0      0 0.0.0.0:50090           0.0.0.0:*               LISTEN      1000       142827      10605/java      
tcp        0      0 0.0.0.0:56779           0.0.0.0:*               LISTEN      1000       142191      10605/java      
tcp        0      0 0.0.0.0:55373           0.0.0.0:*               LISTEN      1000       142251      10696/java      
tcp        0      0 0.0.0.0:50030           0.0.0.0:*               LISTEN      1000     142921      10696/java

I checked the job tracker log file:

2013-01-09 16:40:52,286 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block null bad datanode[0] nodes == null
2013-01-09 16:40:52,286 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/home/hduser/hadoop/tmpnew/mapred/system/jobtracker.info" - Aborting...
2013-01-09 16:40:52,286 WARN org.apache.hadoop.mapred.JobTracker: Writing to file hdfs://master:54310/home/hduser/hadoop/tmpnew/mapred/system/jobtracker.info failed!
2013-01-09 16:40:52,287 WARN org.apache.hadoop.mapred.JobTracker: FileSystem is not ready yet!
2013-01-09 16:40:52,290 WARN org.apache.hadoop.mapred.JobTracker: Failed to initialize recovery manager. 
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /home/hduser/hadoop/tmpnew/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
    at org.apache.hadoop.ipc.Client.call(Client.java:1070)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
    at $Proxy5.addBlock(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
    at $Proxy5.addBlock(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3510)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3373)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2589)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2829)
2013-01-09 16:41:02,291 WARN org.apache.hadoop.mapred.JobTracker: Retrying...
2013-01-09 16:41:02,331 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /home/hduser/hadoop/tmpnew/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)

    at org.apache.hadoop.ipc.Client.call(Client.java:1070)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
    at $Proxy5.addBlock(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
    at $Proxy5.addBlock(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3510)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3373)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2589)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2829)

I am not able to see the tasktracker using the URL - master:50060

Therefore I have a feeling that the tasktracker is not started up correctly.

Help appreciated.

Upvotes: 2

Views: 3992

Answers (2)

stholy
stholy

Reputation: 322

Error persisted from namenode format. Had to setup different user account with fresh installation. Problem solved.

Upvotes: 0

Tariq
Tariq

Reputation: 34184

make sure your HDFS is out of the safemode. if not, use "bin/hadoop dfsadmin -safemode leave" to do that.

Upvotes: 1

Related Questions