Storm UI timeout error

Question

I created 3 aws ec2 servers with RedHat 6 and used this tutorial to deploy storm.

After creating the zookeeper and nimbus instances i could manually start zookeeper and the nimbus/ui nodes. The nimbus:8080 showed me an empty topology. The third server was configured for one supervisor/slave node and i saw it in the ui.

After that I added the supervisord option and changed some ec2 firewall options (unfortunately at the same time).

Now when i start zookeeper, nimbus and ui (with or without supervisord) and look at the ui i get this Error.

java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection timed out

Already tried fiddling around with the aws firewall configs but none is changing anything. Even opening some ports to all ip addresses....

I used this readme to get the right settings.

The logs are all pretty empty. Zookeper seems to accept connections:

2015-12-04 21:47:04,151 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted s2015-12-04 21:47:04,174 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@643] - Established session 0x15170091b370000 with negotiated timeout 20000 for client /52.34.142.187:53935
2015-12-04 21:47:04,177 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@489] - Processed session termination for sessionid: 0x15170091b370000
2015-12-04 21:47:04,179 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for client /52.34.142.187:53935 which had sessionid 0x15170091b370000
2015-12-04 21:47:04,181 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /52.34.142.187:53936
2015-12-04 21:47:04,183 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@898] - Client attempting to establish new session at /52.34.142.187:53936
2015-12-04 21:47:04,187 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@643] - Established session 0x15170091b370001 with negotiated timeout 20000 for client /52.34.142.187:53936
2015-12-04 21:47:04,193 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@489] - Processed session termination for sessionid: 0x15170091b370001
2015-12-04 21:47:04,194 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for client /52.34.142.187:53936 which had sessionid 0x15170091b370001
2015-12-04 21:47:04,201 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /52.34.142.187:53937
2015-12-04 21:47:04,203 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@898] - Client attempting to establish new session at /52.34.142.187:53937
2015-12-04 21:47:04,204 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@643] - Established session 0x15170091b370002 with negotiated timeout 20000 for client /52.34.142.187:53937
2015-12-04 21:47:54,973 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /52.33.187.63:58714
2015-12-04 21:47:55,034 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@898] - Client attempting to establish new session at /52.33.187.63:58714
2015-12-04 21:47:55,035 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@643] - Established session 0x15170091b370003 with negotiated timeout 20000 for client /52.33.187.63:58714
2015-12-04 21:47:56,056 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@489] - Processed session termination for sessionid: 0x15170091b370003
2015-12-04 21:47:56,058 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for client /52.33.187.63:58714 which had sessionid 0x15170091b370003
2015-12-04 21:47:56,063 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /52.33.187.63:58715
2015-12-04 21:47:56,065 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@898] - Client attempting to establish new session at /52.33.187.63:58715
2015-12-04 21:47:56,066 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@643] - Established session 0x15170091b370004 with negotiated timeout 20000 for client /52.33.187.63:58715
2015-12-04 21:49:12,000 [myid:] - INFO  [SessionTracker:ZooKeeperServer@353] - Expiring session 0x15170091b370002, timeout of 20000ms exceeded
2015-12-04 21:49:12,000 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@489] - Processed session termination for sessionid: 0x15170091b370002
2015-12-04 21:49:12,002 [myid:] - INFO  [SyncThread:0:NIOServerCnxn@1008] - Closed socket connection for client /52.34.142.187:53937 which had sessionid 0x15170091b370002
2015-12-04 21:49:14,001 [myid:] - INFO  [SessionTracker:ZooKeeperServer@353] - Expiring session 0x15170091b370004, timeout of 20000ms exceeded
2015-12-04 21:49:14,001 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@489] - Processed session termination for sessionid: 0x15170091b370004
2015-12-04 21:49:14,002 [myid:] - INFO  [SyncThread:0:NIOServerCnxn@1008] - Closed socket connection for client /52.33.187.63:58715 which had sessionid 0x15170091b370004ocket connection from /52.34.142.187:53935
2015-12-04 21:47:04,164 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@898] - Client attempting to establish new session at /52.34.142.187:53935
2015-12-04 21:47:04,165 [myid:] - INFO  [SyncThread:0:FileTxnLog@199] - Creating new log file: log.195

nimbus.log:

2015-12-04 22:11:07.531 o.a.s.s.o.a.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 20082ms for sessionid 0x0, closing socket connection and attempting reconnect
2015-12-04 22:11:08.632 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server ec2-52-34-113-54.us-west-2.compute.amazonaws.com/52.34.113.54:2181. Will not attempt to authenticate using SASL (unknown error)
2015-12-04 22:11:18.481 o.a.s.s.o.a.c.ConnectionState [WARN] Connection attempt unsuccessful after 31043 (greater than max timeout of 20000). Resetting connection and trying again with a new connection.
2015-12-04 22:11:27.743 o.a.s.s.o.a.z.ZooKeeper [INFO] Session: 0x0 closed
2015-12-04 22:11:27.743 o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=zkserver1:2181 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@185100a6
2015-12-04 22:11:27.747 o.a.s.s.o.a.z.ClientCnxn [INFO] EventThread shut down
2015-12-04 22:11:27.750 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server ec2-52-34-113-54.us-west-2.compute.amazonaws.com/52.34.113.54:2181. Will not attempt to authenticate using SASL (unknown error)
2015-12-04 22:11:47.753 o.a.s.s.o.a.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 20006ms for sessionid 0x0, closing socket connection and attempting reconnect
2015-12-04 22:11:48.854 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server ec2-52-34-113-54.us-west-2.compute.amazonaws.com/52.34.113.54:2181. Will not attempt to authenticate using SASL (unknown error)
2015-12-04 22:12:03.860 o.a.s.s.o.a.c.ConnectionState [WARN] Connection attempt unsuccessful after 45379 (greater than max timeout of 20000). Resetting connection and trying again with a new connection.
2015-12-04 22:12:07.958 o.a.s.s.o.a.z.ZooKeeper [INFO] Session: 0x0 closed
2015-12-04 22:12:07.959 o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=zkserver1:2181 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@185100a6
2015-12-04 22:12:07.959 o.a.s.s.o.a.z.ClientCnxn [INFO] EventThread shut down
2015-12-04 22:12:07.960 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server ec2-52-34-113-54.us-west-2.compute.amazonaws.com/52.34.113.54:2181. Will not attempt to authenticate using SASL (unknown error)
2015-12-04 22:12:27.965 o.a.s.s.o.a.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 20006ms for sessionid 0x0, closing socket connection and attempting reconnect
2015-12-04 22:12:29.066 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server ec2-52-34-113-54.us-west-2.compute.amazonaws.com/52.34.113.54:2181. Will not attempt to authenticate using SASL (unknown error)
2015-12-04 22:12:44.074 o.a.s.s.o.a.c.ConnectionState [WARN] Connection attempt unsuccessful after 40213 (greater than max timeout of 20000). Resetting connection and trying again with a new connection.
2015-12-04 22:12:48.169 o.a.s.s.o.a.z.ZooKeeper [INFO] Session: 0x0 closed
2015-12-04 22:12:48.169 o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=zkserver1:2181 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@185100a6
2015-12-04 22:12:48.170 o.a.s.s.o.a.z.ClientCnxn [INFO] EventThread shut down
2015-12-04 22:12:48.171 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server ec2-52-34-113-54.us-west-2.compute.amazonaws.com/52.34.113.54:2181. Will not attempt to authenticate using SASL (unknown error)
2015-12-04 22:13:03.172 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to ec2-52-34-113-54.us-west-2.compute.amazonaws.com/52.34.113.54:2181, initiating session
2015-12-04 22:13:03.177 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server ec2-52-34-113-54.us-west-2.compute.amazonaws.com/52.34.113.54:2181, sessionid = 0x15170091b370005, negotiated timeout = 20000
2015-12-04 22:13:03.179 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-12-04 22:13:03.180 b.s.zookeeper [INFO] Zookeeper state update: :connected:none
2015-12-04 22:13:03.186 o.a.s.s.o.a.z.ZooKeeper [INFO] Session: 0x15170091b370005 closed
2015-12-04 22:13:03.186 o.a.s.s.o.a.z.ClientCnxn [INFO] EventThread shut down
2015-12-04 22:13:03.187 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
2015-12-04 22:13:03.188 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-12-04 22:13:03.190 o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=zkserver1:2181/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@6c4fb026
2015-12-04 22:13:03.191 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server ec2-52-34-113-54.us-west-2.compute.amazonaws.com/52.34.113.54:2181. Will not attempt to authenticate using SASL (unknown error)
2015-12-04 22:13:03.193 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to ec2-52-34-113-54.us-west-2.compute.amazonaws.com/52.34.113.54:2181, initiating session
2015-12-04 22:13:03.195 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server ec2-52-34-113-54.us-west-2.compute.amazonaws.com/52.34.113.54:2181, sessionid = 0x15170091b370006, negotiated timeout = 20000
2015-12-04 22:13:03.195 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED  
2015-12-04 22:13:03.233 b.s.d.nimbus [INFO] Starting Nimbus server...

Any ideas?

Answer 1

i found out... my hosts file was global, so the nimbus ip was the external ip of my machine. It tried to connect through that ip even though it was on localhost, the firewall needs to configured properly or the hosts file adjusted