CODEWITHSUNDEEP
linuxhadoopbigdatahbaseapache-zookeeper
MrWrzosek
MrWrzosek

Reputation: 201

HBase Shell - org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet

I am trying to set up distributed HBase on 3 nodes. I have already set up hadoop, YARN ZooKeeper and now HBase but when I launch hbase shell and run the simplest command for example status or list I get the exception:

HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell
Version 2.4.8, rf844d09157d9dce6c54fcd53975b7a45865ee9ac, Wed Oct 27 08:48:57 PDT 2021
Took 0.0014 seconds
hbase:001:0> status

ERROR: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
        at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2817)
        at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:1205)
        at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:392)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:354)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:334)

For usage try 'help "status"'

Took 9.0800 seconds

My Hadoop Master JPS:

3443 NameNode
13987 Jps
2503 QuorumPeerMain
3754 SecondaryNameNode
4010 ResourceManager
4749 HMaster

and both Nodes JPS:

2135 QuorumPeerMain
6327 HRegionServer
3115 NodeManager
2894 DataNode
6670 Jps

I launch my own ZooKeeper with following zoo.cfg:

tickTime=2000
dataDir=/usr/local/zookeeper/data
dataLogDir=/usr/local/zookeeper/data
clientPort=2181
initLimit=10
syncLimit=10
server.1=0.0.0.0:2888:3888
server.2=OpenCitation2:2888:3888
server.3=OpenCitation3:2888:3888
4lw.commands.whitelist=stat, ruok, conf, isro



hadoopuser@OpenCitation1:~$ echo ruok | nc OpenCitation1 2181
imok
hadoopuser@OpenCitation1:~$ echo ruok | nc OpenCitation2 2181
imok
hadoopuser@OpenCitation1:~$ echo ruok | nc OpenCitation3 2181
imok

Hbase-site.xml:

      <property>
        <name>hbase.rootdir</name>
        <value>hdfs://OpenCitation1:9000/hbase</value>
      </property>
      <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
      </property>
      <property>
        <name>hbase.zookeeper.property.dataDir</name>
        <value>/usr/local/zookeeper/data</value>
      </property>
      <property>
        <name>hbase.unsafe.stream.capability.enforce</name>
        <value>false</value>
      </property>
      <property>
        <name>hbase.tmp.dir</name>
        <value>/usr/local/HBase/var</value>
      </property>
    <property>
        <name>hbase.zookeeper.property.clientPort</name>
        <value>2181</value>
    </property>

Hbase-env.sh:

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
export HBASE_MANAGES_ZK=false

Hadoop version:

hadoopuser@OpenCitation1:~$ hadoop version
Hadoop 3.3.1
Source code repository https://github.com/apache/hadoop.git -r a3b9c37a397ad4188041dd80621bdeefc46885f2
Compiled by ubuntu on 2021-06-15T05:13Z
Compiled with protoc 3.7.1
From source with checksum 88a4ddb2299aca054416d6b7f81ca55
This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-3.3.1.jar

Hbase version:

hbase:001:0> version
2.4.8, rf844d09157d9dce6c54fcd53975b7a45865ee9ac, Wed Oct 27 08:48:57 PDT 2021
Took 0.0005 seconds

Also I tried turning safemode:

hadoopuser@OpenCitation1:~$ hdfs dfsadmin -safemode get
Safe mode is OFF

but it didn't work either.

I need help because I am really running out od ideas...

Thanks in advance

Upvotes: 4

Views: 5629

Answers (2)

Jannick Kappelmann
Jannick Kappelmann

Reputation: 34

To have a running HMaster process according to jps is not sufficient for the Hbase Master to work properly.

The Hbase Master was propably still initializing. You can see the status in the Hbase GUI https://Hbase master address:16010

For as long as the HMaster is shown to be not ready, all interactions will fail with your above error message.

For instance, when the below state is shown in the GUI, you will get the org.apache.hadoop.hbase.ipc.ServerNotRunningYetException Exception.

Start Time Description State Status Completion Time
Sat Apr 15 17:17:36 UTC 2023 Master startup RUNNING (since 15mins, 47sec ago) Initialize ServerManager and schedule SCP for crash servers (since 15mins, 47sec ago) RUNNING

Upvotes: 0

MrWrzosek
MrWrzosek

Reputation: 201

UPDATE:

I have solved the issue by adding the following property to the hbase-site.xml:

<property>
  <name>hbase.wal.provider</name>
  <value>filesystem</value>
</property>

Now it runs:

hadoopuser@OpenCitation1:~$ hbase shell
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell
Version 2.4.8, rf844d09157d9dce6c54fcd53975b7a45865ee9ac, Wed Oct 27 08:48:57 PDT 2021
Took 0.0014 seconds
hbase:001:0> status
1 active master, 0 backup masters, 2 servers, 0 dead, 1.0000 average load
Took 0.4730 seconds
hbase:002:0>

Maybe it will help someone

Upvotes: 15

Related Questions