Hbase Regions server is not able to communicate with HMaster

Question

I am not able to setup the hbase in distributed mode. It works fine when i setup it on one machine(standalone mode). My Zookeeper, hmaster and region server starts properly.
But when i go to hbase shell and look for the status. It shows me 0 region server. I am attaching my logs of regions server. Plus the host files of my master(namenode) and slave(datanode). I have tried every P&C which are given on stackoverflow for changing the host file, but didn't work for me.

       2013-06-24 15:03:45,844 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server naresh-pc/192.168.0.108:2181. Will not attempt to authenticate using SASL (unknown error)
            2013-06-24 15:03:45,845 WARN org.apache.zookeeper.ClientCnxn: Session 0x13f75807d960001 for server null, unexpected error, closing socket connection and attempting to reconnect

            java.net.ConnectException: Connection refused
                    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
                    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
                    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
                    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

     Slave /etc/hosts :

    127.0.0.1       localhost
    127.0.1.1       ubuntu-pc
    #ip for hadoop
    192.168.0.108   master
    192.168.0.126   slave

    # The following lines are desirable for IPv6 capable hosts
    ::1     ip6-localhost ip6-loopback
    fe00::0 ip6-localnet
    ff00::0 ip6-mcastprefix
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters

    Master /etc/hosts :

    127.0.0.1       localhost
    127.0.1.1       naresh-pc
   #ip for hadoop 
    192.168.0.108   master
    192.168.0.126   slave


    # The following lines are desirable for IPv6 capable hosts
    ::1     ip6-localhost ip6-loopback
    fe00::0 ip6-localnet
    ff00::0 ip6-mcastprefix
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters

hbase-site.xml :

  <configuration>  
 <property>
    <name>hbase.master</name>
           <value>master:60000</value>
          <description>The host and port that the HBase master runs at.
                    A value of 'local' runs the master and a regionserver
                                                     in a single process.
                               </description>
                </property>

                <property>
                                <name>hbase.rootdir</name>
                                <value>hdfs://master:54310/hbase</value>
                                <description>The directory shared by region servers.</description>
                </property>


<property>
                                <name>hbase.cluster.distributed</name>
                                <value>true</value>
                                <description>The mode the cluster will be in. Possible values are
                                false: standalone and pseudo-distributed setups with managed
                                Zookeeper true: fully-distributed with unmanaged Zookeeper
                                Quorum (see hbase-env.sh)
                                </description>
                </property>
                <property>
                                <name>hbase.zookeeper.property.clientPort</name>
                                <value>2181</value>
                                <description>Property from ZooKeeper's config zoo.cfg.
                                The port at which the clients will connect.
                                </description>
                </property>

                <property>
           <name>hbase.zookeeper.quorum</name>
             <value>master</value>
                <description>Comma separated list of servers in the ZooKeeper Quorum.
                                     For example,
                                     "host1.mydomain.com,host2.mydomain.com".
                                     By default this is set to localhost for local and
                                     pseudo-distributed modes of operation. For a
                                     fully-distributed setup, this should be set to a full
                                     list of ZooKeeper quorum servers. If
                                     HBASE_MANAGES_ZK is set in hbase-env.sh
                                     this is the list of servers which we will start/stop
                                     ZooKeeper on.
                </description>
                </property>
</configuration>

    Zookeeper log:

         2013-06-28 18:22:26,781 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception
    EndOfStreamException: Unable to read additional data from client sessionid 0x13f8ac0b91b0002, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
        at java.lang.Thread.run(Thread.java:722)
    2013-06-28 18:22:26,858 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /192.168.0.108:57447 which had sessionid 0x13f8ac0b91b0002
    2013-06-28 18:25:21,001 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x13f8ac0b91b0002, timeout of 180000ms exceeded
    2013-06-28 18:25:21,002 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x13f8ac0b91b0002      

Master Log:
2013-06-28 18:22:41,932 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting for region servers count to settle; currently checked in 0, slept for 1502022 ms
2013-06-28 18:22:43,457 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting for region servers count to settle; currently checked in 0, slept for 1503547 ms

Answer 1

Can you try all of this:

Make sure your /conf/regionservers file has just one entry: slave

Not sure what HBase version you are using, but instead of using port 54310 for hbase.rootdir property in your hbase-site.xml use port 9000

Your /etc/hosts file, on BOTH master and slave should should only have these custom entries:

127.0.0.1 localhost
192.168.0.108 master
192.168.0.126 slave

I am concerned that your logs state Opening socket connection to server naresh-pc/192.168.0.108:2181

Clearly the system thinks that zookeeper is on host naresh-pc, but in your config you are setting zookeeper quorum at host master, which HBase will bind to. That's a problem right there. In my experience, HBase is EXTREMELY fussy about host names, so make sure they are all in synch in all your configs and in your /etc/hosts file.

Also, this may be a minor issue, but wouldn't hurt to specify the zookeper data directory in your .xml file to have a minimum set of settings that should make the cluster work: hbase.zookeeper.property.dataDir

Answer 2

Your Regionserver is looking for HMaster at naresh-pc but you do not have any such entry in your /etc/hosts file. Please make sure your configuration is proper.

Answer 3

Remove 127.0.1.1 from hosts file and turn of IPV6. That should fix the problem.