hBase - the server has too many connections

Question

I use hbase to store my data which comes from a crawler. These data are indexed from hbase to Solr. But I have a problem with hBase. As I have larger DB now, everytime I try to index data from hbase the error shown below occures. This is the part of log file hadoop.log

There is something like Caused by: This could be a sign that the server has too many connections (30 is the default). and I found here on SO that this could be beacues of maxClientConnections property so I set up this value to 0 but this solution did not help.

2013-11-11 15:30:22,638 ERROR zookeeper.ClientCnxn - Error while calling watcher 
java.lang.NullPointerException
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:315)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1340)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:343)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:261)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:488)
2013-11-11 15:30:27,925 WARN  mapred.FileOutputCommitter - Output path is null in cleanup
2013-11-11 15:30:27,925 WARN  mapred.LocalJobRunner - job_local1497313370_0001
org.apache.gora.util.GoraException: java.lang.RuntimeException: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:118)
at org.apache.gora.mapreduce.GoraOutputFormat.getRecordWriter(GoraOutputFormat.java:88)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:568)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:637)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:398)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
at org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:127)
at org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
... 6 more
Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:155)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1002)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:304)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:295)
at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:157)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:90)
at org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:109)
... 8 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:903)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
... 14 more

Do you know where is the problem please? Or do you know how can I make sure I have not opened multiple HTable instances simultaneously?

Answer 1

HTable does not open new connections, HTable resides on Connection object. so many HTables are sharing the same connection. but you do have to make sure you don't open too many connections per node.

ZK has a setting to set max connections from one IP address, to prevent connection bombing. default is 30. you can increase that but not by too much. normally there shouldn't be too many concurrent connections to ZK from the same host. otherwise use netstat to inspect what process is opening too many ZK connections, that process might have connection leakage.

refer to my other related answer: Error HBASE-ZOOKEEPER : Too many connections

Answer 2

If somebody is interested how I figured out this problem, I created a thread in hBase mailing-list and solved it there http://apache-hbase.679495.n3.nabble.com/hBase-the-server-has-too-many-connections-maxClientConn-property-set-to-0-does-not-help-td4052728.html#a4052873

I don't know what exactly solved the problem but I tried to combine these things:

1) property maxClientConnections set up to:

<property>
   <name>hbase.zookeeper.property.maxClientCnxns</name>
   <value>0</value>
</property>

2) increased time out

<property>
  <name>zookeeper.session.timeout</name>
  <value>1200000</value>
</property>

<property>
  <name>hbase.zookeeper.property.tickTime</name>
  <value>6000</value>
</property>

3) https://wiki.apache.org/nutch/ErrorMessagesInNutch2#Nutch_2.1_.2B-_HBase_0.90.4_cluster_settings_-WARN_zookeeper.ClientCnxn_-_Session_0x0_for_server_node1.xxxxxx.com.2Fxxx.xxx.xxx.xxx:2181.2C_unexpected_error.2C_closing_socket_connection_and_attempting_reconnect_java.io.IOException:_Connection_reset_by_peer

Then I killed all java processes, start and stop hBase and then run rolling-restart.sh and everything works fine now.