Apache Storm tries localhost zookeeper connection instead of specified list of zookeeper servers

Question

Storm 1.0.3
URL https://git-wip-us.apache.org/repos/asf/storm.git -r eac433b0beb3798c4723deb39b3c4fad446378f4
Branch (no branch)
Compiled by ptgoetz on 2017-02-07T20:22Z
From source with checksum c78e52de4b8a22d99551d45dfe9c1a4b

[root@storm-nimbus-prod]/opt/storm/logs# java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

env

export STORM_HOME=/opt/storm
export STORM=/opt/storm/bin
export STORM_CONFIG=/opt/storm/conf

conf/storm.yaml

storm.zookeper.servers:
  - "10.138.0.8"
  - "10.138.0.9"
  - "10.138.0.16"

storm.zookeeper.port: 2181

nimbus.seeds: ["localhost"]

storm.local.dir: "/var/log/storm"
supervisor.slots.ports:
  - 6700
  - 6701
  - 6702
  - 6703

worker.childopts: "-Xmx768m"
nimbus.childopts: "-Xmx512m"
supervisor.childopts: "-Xmx256m"

The problem

When I run ./storm nimbus I'm getting weird messages in opt/storm/logs/nimbus.log

2017-04-02 05:21:26.782 main o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2017-04-02 05:21:26.789 main o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=localhost:2181 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@10afe71a
2017-04-02 05:21:26.802 main-SendThread(localhost:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-04-02 05:21:26.805 main-SendThread(localhost:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-04-02 05:21:26.808 main-SendThread(localhost:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-04-02 05:21:26.892 main-SendThread(localhost:2181) o.a.s.s.o.a.z.ClientCnxn [WARN] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at org.apache.storm.shade.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
        at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2017-04-02 05:23:03.252 main-SendThread(localhost:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-04-02 05:23:03.353 main o.a.s.s.o.a.z.ZooKeeper [INFO] Session: 0x0 closed
2017-04-02 05:23:03.353 main o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=localhost:2181 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@10afe71a
2017-04-02 05:23:03.354 main-EventThread o.a.s.s.o.a.z.ClientCnxn [INFO] EventThread shut down
2017-04-02 05:23:03.356 main-SendThread(localhost:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-04-02 05:23:03.356 main-SendThread(localhost:2181) o.a.s.s.o.a.z.ClientCnxn [WARN] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused

so it seems it tries to connect to localhost:2181 for Zookeeper, while I have listed 3 IPs I can't find in logs at all.

I don't have local zookeeper up&running, but all 3 Zookeeper IPs are OK:

[root@storm-nimbus-prod]/opt/storm/logs# echo stat | nc 10.138.0.8 2181    
Zookeeper version: 3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
Clients:
 /10.138.0.10:60690[0](queued=0,recved=1,sent=0)
 /10.138.0.14:44242[1](queued=0,recved=4421,sent=4421)

Latency min/avg/max: 0/0/24
Received: 5268
Sent: 5272
Connections: 2
Outstanding: 0
Zxid: 0xf00000193
Mode: leader
Node count: 54
[root@storm-nimbus-prod]/opt/storm/logs# echo stat | nc 10.138.0.9 2181
Zookeeper version: 3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
Clients:
 /10.138.0.13:51812[1](queued=0,recved=4536,sent=4543)
 /10.138.0.10:44096[0](queued=0,recved=1,sent=0)

Latency min/avg/max: 0/0/18
Received: 5310
Sent: 5320
Connections: 2
Outstanding: 0
Zxid: 0xf00000193
Mode: follower
Node count: 54
[root@storm-nimbus-prod]/opt/storm/logs# echo stat | nc 10.138.0.16 2181
Zookeeper version: 3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
Clients:
 /10.138.0.10:34488[0](queued=0,recved=1,sent=0)

Latency min/avg/max: 0/0/18
Received: 973
Sent: 984
Connections: 1
Outstanding: 0
Zxid: 0xf00000193
Mode: follower
Node count: 54

Any ideas?

Answer 1

Ok the problem was that storm.zookeper.servers missed 'e', it should have been

storm.zookeeper.servers