Apache Zookeeper multi-node communication error

Question

I have three znode: Apache Zookeeper, 3.4.8 JAVA 1.8_77 RedHat 6.7, Selinux disabled, Firewall disabled, IPV6 disabled

Hosts:

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.0.133 v175
192.168.0.134 v176
192.168.0.125 V177

Config:

tickTime=2000
dataDir=/home/znode/datadir
clientPort=2181
initLimit=5
syncLimit=2
server.1=v175:2888:3888
server.2=v176:2888:3888
server.3=v177:2888:3888

Errors:

essage format version), 2 (n.leader), 0x0 (n.zxid), 0x100f (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LEADING (my state)
2016-04-05 16:26:00,270 [myid:3] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x100f (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LEADING (my state)
2016-04-05 16:26:03,099 [myid:3] - WARN  [QuorumPeer[myid=3]/0.0.0.0:2181:QuorumPeer@862] - Unexpected exception
java.lang.InterruptedException: Timeout while waiting for epoch from quorum
        at org.apache.zookeeper.server.quorum.Leader.getEpochToPropose(Leader.java:881)
        at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:380)
        at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:859)
2016-04-05 16:26:03,100 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:Leader@496] - Shutting down
2016-04-05 16:26:03,100 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:Leader@502] - Shutdown called
java.lang.Exception: shutdown Leader! reason: Forcing shutdown
        at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:502)
        at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:865)
2016-04-05 16:26:03,100 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:QuorumPeer@774] - LOOKING
2016-04-05 16:26:03,100 [myid:3] - INFO  [LearnerCnxAcceptor-V177/192.168.0.125:2888:Leader$LearnerCnxAcceptor@325] - exception while shutting down acceptor: java.net.SocketException: Socket closed
2016-04-05 16:26:03,100 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:FastLeaderElection@818] - New election. My id =  3, proposed zxid=0x0
2016-04-05 16:26:03,102 [myid:3] - WARN  [WorkerSender[myid=3]:QuorumCnxManager@400] - Cannot open channel to 2 at election address v176/192.168.0.134:3888
java.net.NoRouteToHostException: No route to host
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
        at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
        at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
        at java.lang.Thread.run(Thread.java:745)
2016-04-05 16:26:03,103 [myid:3] - INFO  [WorkerSender[myid=3]:QuorumPeer$QuorumServer@149] - Resolved hostname: v176 to address: v176/192.168.0.134
2016-04-05 16:26:03,102 [myid:3] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x100f (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)

Servers cannot communicate with each other. Help !

roguequery · Accepted Answer

if you netstat -tulnap the server, are ports 2888, 3888 ad 2181 open and listening or can only localhost (0.0.0.0) on each of those boxes hit 2181, 288, and 3888?

EDIT:

Looking at your netstat I see this:

tcp 0 0 192.168.0.125:2888 0.0.0.0:* LISTEN
tcp 0 0 192.168.0.125:3888

means you need to modify your etc/hots to use 0.0.0.0 with the hostname.

So if the box's hostname is zoobox1 the /etc/hosts needs to have this line in it:

127.0.0.1 localhost
0.0.0.0 zoobox1

this way the netstat -tulnap will open port 2888 and 3888 so other servers then localhost can connect.

Apache Zookeeper multi-node communication error

Answers (2)

Related Questions