Zookeeper: started but it is probably not running

Question

I'm currently using zookeeper for stuff, and I did run it easily several times. but it seem my cluster nodes has some issues with network connection yesterday , so I reboot the system. when I run : bin/zkServer.sh: ,it says STARTED. but when I want to get status: zookeeper probably not running and if I start it again it says : zookeeper already running with pid = 123

is there any problem? how can I fix this? thank you.

ps: here is the log when I start-foreground:

ZooKeeper JMX enabled by default
Using config: /home/mesto/zookeeper-3.4.10/bin/../conf/zoo.cfg
2018-03-11 12:47:51,908 [myid:] - INFO  [main:QuorumPeerConfig@134] -Reading configuration from: /home/mesto/zookeeper-3.4.10/bin/../conf/zoo.cfg
2018-03-11 12:47:51,920 [myid:] - INFO  [main:QuorumPeer$QuorumServer@167] - Resolved hostname: 11.0.0.1 to address: /11.0.0.1
2018-03-11 12:47:51,921 [myid:] - INFO  [main:QuorumPeer$QuorumServer@167] - Resolved hostname: 11.0.0.3 to address: /11.0.0.3
2018-03-11 12:47:51,921 [myid:] - INFO  [main:QuorumPeer$QuorumServer@167] - Resolved hostname: 11.0.0.2 to address: /11.0.0.2
2018-03-11 12:47:51,921 [myid:] - INFO  [main:QuorumPeerConfig@396] - Defaulting to majority quorums
2018-03-11 12:47:51,939 [myid:1] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2018-03-11 12:47:51,939 [myid:1] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2018-03-11 12:47:51,939 [myid:1] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2018-03-11 12:47:51,962 [myid:1] - INFO  [main:QuorumPeerMain@127] - Starting quorum peer
2018-03-11 12:47:51,969 [myid:1] - INFO  [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181
2018-03-11 12:47:51,982 [myid:1] - INFO  [main:QuorumPeer@1134] - minSessionTimeout set to -1
2018-03-11 12:47:51,982 [myid:1] - INFO  [main:QuorumPeer@1145] - maxSessionTimeout set to -1
2018-03-11 12:47:51,982 [myid:1] - INFO  [main:QuorumPeer@1419] - QuorumPeer communication is not secured!
2018-03-11 12:47:51,982 [myid:1] - INFO  [main:QuorumPeer@1448] - quorum.cnxn.threads.size set to 20
2018-03-11 12:47:52,010 [myid:1] - INFO  [ListenerThread:QuorumCnxManager$Listener@739] - My election bind port: /11.0.0.1:3888
2018-03-11 12:47:52,016 [myid:1] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer@865] - LOOKING
2018-03-11 12:47:52,018 [myid:1] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@818] - New election. My id =  1, proposed zxid=0x0
2018-03-11 12:47:52,020 [myid:1] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-03-11 12:47:57,026 [myid:1] - WARN  [WorkerSender[myid=1]:QuorumCnxManager@588] - Cannot open channel to 2 at election address /11.0.0.2:3888
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:562)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:538)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
at java.lang.Thread.run(Thread.java:745)
2018-03-11 12:47:57,030 [myid:1] - INFO  [WorkerSender[myid=1]:QuorumPeer$QuorumServer@167] - Resolved hostname: 11.0.0.2 to address: /11.0.0.2
2018-03-11 12:48:02,036 [myid:1] - WARN  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@588] - Cannot open channel to 2 at election address /11.0.0.2:3888

Answer 1

Zookeeper is running, but no node can be elected as leader for some reason. The output of ./sbin/zkServer.sh status is not proper.

There is a way to confirm that, run :

echo srvr |nc localhost 2181

You may see the result like:

This ZooKeeper instance is not currently serving requests

The solution is to start the other two nodes, and check if the election is done.