CODEWITHSUNDEEP
apache-zookeeperapache-storm
numb3rs1x
numb3rs1x

Reputation: 5273

storm dies when certain zookeeper nodes fail

I have a nimbus server and 3 zookeeper nodes.

My storm.yaml file looks like this:

storm.zookeeper.servers:
 - "server1"
 - "server2"
 - "server3"

nimbus.host: "nimbus-server"

storm.local.dir: "/var/storm"

My zoo.cfg files all look like this:

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/zookeeper
clientPort=2181
autopurge.snapRetainCount=3
autopurge.purgeInterval=1
server.3=server1:2888:3888
server.4=server2:2888:3888
server.5=server3:2888:3888

When all three zookeeper nodes are running, everything is fine according to the storm_ui. If I shut down one of these three nodes, the nimbus server complains that it can't connect to the zookeeper cluster and it dies. I can't find anywhere why this might be happening. The documentation says that if I have three zookeeper nodes, it should tolerate one of them dying. Is there something that has to be set in one of these for this to work?

Upvotes: 2

Views: 736

Answers (1)

numb3rs1x
numb3rs1x

Reputation: 5273

This turned out to be iptables. There never was a quorum between the zookeeper servers, so in effect, after the one I stopped was out, it behaved just as it should have. I opened port 2181, 2888, and 3888 on the one server that didn't have them opened and now I can kill one of them with storm still alive.

Upvotes: 2

Related Questions