org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /lock/test3_node

Question

I use Curator client to operate zookeeper server, but always failed if I start ten threads to operate zookeeper server concurrently.

It works fine if the threads number less than 6, while the connection lose occurs, I　must restart the zookeeper server. somebody knows how to resolve this issue?

curatorClient = CuratorFrameworkFactory.builder().connectString(zkAddress).sessionTimeoutMs(Constant.ZK_SESSION_TIMEOUT).connectionTimeoutMs(Constant.ZK_CONNECTION_TIMEOUT).retryPolicy(retryPolicy).build();
curatorClient.start();

I did the following operation: curatorClient.create().inbackground(new callback(){}).forPath(); curatorClient.delete().inbackground(new callback(){}).forPath(); lock.acquire();..

ERROR:

org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode    
= ConnectionLoss for /lock/test3_node
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1500)
at  

org.apache.curator.framework.imps.CreateBuilderImpl$12.call(CreateBuilderImpl.java:766)
at org.apache.curator.framework.imps.CreateBuilderImpl$12.call(CreateBuilderImpl.java:758)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
at org.apache.curator.framework.imps.CreateBuilderImpl.findProtectedNodeInForeground(CreateBuilderImpl.java:754)
at org.apache.curator.framework.imps.CreateBuilderImpl.access$1300(CreateBuilderImpl.java:44)
at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:713)
at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:703)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:699)
at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:477)
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:467)
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:447)
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
at org.apache.curator.framework.recipes.locks.StandardLockInternalsDriver.createsTheLock(StandardLockInternalsDriver.java:54)
at org.apache.curator.framework.recipes.locks.LockInternals.attemptLock(LockInternals.java:216)
at org.apache.curator.framework.recipes.locks.InterProcessMutex.internalLock(InterProcessMutex.java:232)
at org.apache.curator.framework.recipes.locks.InterProcessMutex.acquire(InterProcessMutex.java:108)
at 
$InsertQueueCallBack.processResult(CuratorUtil.java:614)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:743)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:520)
at org.apache.curator.framework.imps.CreateBuilderImpl.sendBackgroundResponse(CreateBuilderImpl.java:565)
at org.apache.curator.framework.imps.CreateBuilderImpl.access$900(CreateBuilderImpl.java:44)
at org.apache.curator.framework.imps.CreateBuilderImpl$6.processResult(CreateBuilderImpl.java:524)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
.....
Connection timed out for connection string (slc03nwh.us.oracle.com:2181) and timeout (15000) / elapsed (5380)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:195)
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:87)
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:821)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:807)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:63)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Then I add ConnectionStateListener and UnhandledErrorListener, I got the following information:

 newState.isConnected()false
 SUSPENDED
 newState.isConnected()true
 RECONNECTED
 newState.isConnected()lost
 RECONNECTED

  unhandledErrorListener:Background operation retry gave up
  Background retry gave up

Answer 1

I tried to use singleton instance to create client, and set the connection timeout longer. this issue was gone.

But I am not find the root reason to cause connection lose error.