HMaster gets stopped soon after startup

Question

I have read many similar questions and tried to resolve but not succeeded yet. I have a hadoop 2.6.0 cluster (one namenode at x.x.x.55 and 3 datanodes at x.x.x.54, 56, 57). Then I installed hbase-0.98.12-hadoop2 on the cluster. In hbase-env.sh I uncomment HBASE_MANAGES_ZK = true, and set Java home path. Here is the hbase-site.xml:

<property>
    <name>hbase.master</name>
    <value>hadoop.master:60000</value>
</property>
<property>
    <name>hbase.rootdir</name>
    <value>hdfs://hadoop.master:9000/hbase</value>
</property>
<property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
</property>
<property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2222</value>
</property>
<property>
    <name>hbase.zookeeper.quorum</name>
    <value>hadoop.master</value>
</property>

Right after I started hbase, jps printed out:

5960 NameNode
7635 HMaster
8062 Jps
6154 SecondaryNameNode
7537 HQuorumPeer

But very soon later (1 or 2 mins), HMaster got stopped. Here is Hmaster logs:

2015-06-22 15:47:13,286 INFO  [master:hadoop:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 3, slept for 856 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-06-22 15:47:14,803 INFO  [master:hadoop:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 3, slept for 2373 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-06-22 15:47:16,315 INFO  [master:hadoop:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 3, slept for 3885 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-06-22 15:47:16,974 INFO  [master:hadoop:60000] master.ServerManager: Finished waiting for region servers count to settle; checked in 3, slept for 4544 ms, expecting minimum of 1, maximum of 2147483647, master is running.
2015-06-22 15:47:16,994 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.master,60020,1434939246333 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,994 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave1,16020,1434708308676-splitting doesn't belong to a known region server, splitting
2015-06-22 15:47:16,995 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave1,16020,1434935680372 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,995 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave1,60020,1434939607069 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,996 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave1,60020,1434940811464 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,996 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave1,60020,1434941688620 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,996 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave1,60020,1434950660701 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,996 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave1,60020,1434952134591 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,996 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave1,60020,1434955629747 belongs to an existing region server
2015-06-22 15:47:16,997 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave2,16020,1434708308695-splitting doesn't belong to a known region server, splitting
2015-06-22 15:47:16,997 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave2,16020,1434935679919 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,997 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave2,60020,1434939606935 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,997 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave2,60020,1434940810938 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,997 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave2,60020,1434941688163 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,997 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave2,60020,1434950660936 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,997 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave2,60020,1434952134902 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,998 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave2,60020,1434955629669 belongs to an existing region server
2015-06-22 15:47:16,998 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave3,16020,1434935540949 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,998 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave3,16020,1434935680550-splitting doesn't belong to a known region server, splitting
2015-06-22 15:47:16,999 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave3,60020,1434939607523 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,999 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave3,60020,1434940811989 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,999 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave3,60020,1434941689997 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,999 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave3,60020,1434950660287 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,999 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave3,60020,1434955629745 belongs to an existing region server
2015-06-22 15:47:17,000 FATAL [master:hadoop:60000] master.HMaster: Master server abort: loaded coprocessors are: []
2015-06-22 15:47:17,002 FATAL [master:hadoop:60000] master.HMaster: Unhandled exception. Starting shutdown.
java.lang.IllegalArgumentException: Not a host:port pair: hregion-79918053
at org.apache.hadoop.hbase.util.Addressing.parseHostname(Addressing.java:62)
at org.apache.hadoop.hbase.ServerName.<init>(ServerName.java:128)
at org.apache.hadoop.hbase.ServerName.valueOf(ServerName.java:177)
at org.apache.hadoop.hbase.ServerName.parseServerName(ServerName.java:347)
at org.apache.hadoop.hbase.master.MasterFileSystem.getFailedServersFromLogFolders(MasterFileSystem.java:253)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:944)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:683)
at java.lang.Thread.run(Thread.java:745)
2015-06-22 15:47:17,006 INFO  [master:hadoop:60000] master.HMaster: Aborting
2015-06-22 15:47:17,007 DEBUG [master:hadoop:60000] master.HMaster: Stopping service threads
2015-06-22 15:47:17,007 INFO  [master:hadoop:60000] ipc.RpcServer: Stopping server on 60000
2015-06-22 15:47:17,007 INFO  [RpcServer.listener,port=60000] ipc.RpcServer: RpcServer.listener,port=60000: stopping
2015-06-22 15:47:17,010 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2015-06-22 15:47:17,011 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2015-06-22 15:47:17,013 INFO  [master:hadoop:60000.oldLogCleaner] cleaner.LogCleaner: master:hadoop:60000.oldLogCleaner exiting
2015-06-22 15:47:17,013 INFO  [master:hadoop:60000.oldLogCleaner] master.ReplicationLogCleaner: Stopping replicationLogCleaner-0x14e1a05351e0005, quorum=hadoop.master:2222, baseZNode=/hbase
2015-06-22 15:47:17,014 INFO  [master:hadoop:60000.archivedHFileCleaner] cleaner.HFileCleaner: master:hadoop:60000.archivedHFileCleaner exiting
2015-06-22 15:47:17,016 INFO  [master:hadoop:60000] master.HMaster: Stopping infoServer
2015-06-22 15:47:17,017 INFO  [master:hadoop:60000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-06-22 15:47:17,017 INFO  [master:hadoop:60000.oldLogCleaner] zookeeper.ZooKeeper: Session: 0x14e1a05351e0005 closed
2015-06-22 15:47:17,024 INFO  [master:hadoop:60000] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60010
2015-06-22 15:47:17,031 DEBUG [master:hadoop:60000] catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@71e15889
2015-06-22 15:47:17,031 INFO  [master:hadoop:60000] client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14e1a05351e0004
2015-06-22 15:47:17,034 INFO  [master:hadoop:60000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-06-22 15:47:17,034 INFO  [master:hadoop:60000] zookeeper.ZooKeeper: Session: 0x14e1a05351e0004 closed
2015-06-22 15:47:17,034 INFO  [hadoop.master,60000,1434955628046.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor: hadoop.master,60000,1434955628046.splitLogManagerTimeoutMonitor exiting
2015-06-22 15:47:17,037 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-06-22 15:47:17,037 INFO  [master:hadoop:60000] zookeeper.ZooKeeper: Session: 0x14e1a05351e0000 closed
2015-06-22 15:47:17,037 INFO  [master:hadoop:60000] master.HMaster: HMaster main thread exiting
2015-06-22 15:47:17,038 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: HMaster Aborted
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:201)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3029)

Note: all nodes use the same hbase-0.98.12.

Please help to resolve this.

Answer 1

I was getting this error after reverting from hbase 1.1.2 to hbase 0.98 because I forget to delete /hbase in hdfs.

Answer 2

You have java.lang.IllegalArgumentException: Not a host:port pair: exception. This arises mostly due to incompatible versions of hbase with other componenets.

You can have a look at this compatibility matrix, your hadoop 2.6.0 cluster with hbase 0.98 is not tested.

http://hbase.apache.org/book.html#hadoop