CODEWITHSUNDEEP
nginxwebsocketsocket.iohaproxy
Ilya
Ilya

Reputation: 627

HAProxy + Socket.IO + TornadIO Constant Disconnects on Heartbeat

So, I am having trouble load balancing socket.io on port 8888 using HAProxy. My setup is NGINX listening on port 80, and load balancing between Tornado web server instances running on port 80. Then, on the same load balancer, I have a HAProxy instance listening on port 8888, forwarding requests to OTHER computers in the network hosting TornadIO server instances also running on 8888. The connection works for the first 30 seconds or so, and then begins to disconnect / reconnect repeatedly. What's important to notice is that it seems like it breaks on the first heartbeat attempt ... is the heartbeat a different protocol that HAProxy would have trouble with as opposed to the first connection attempt / first few messages exchanged?

Interestingly, this DOES NOT happen when the tornadIO instance is running on the same computer as the load balancer, even with HAProxy working (but connecting port 8888 and lets say the tornadIO instance on port 9000).

It's important to note that TornadIO does not throw any exceptions, or any upset output during this entire process, showing that its not my server code but something in the proxy layer?

Let it also be known that I am using RabbitMQ to synchronize all the TonadIO clusters, not that I think it matters (and HAProxy does not touch Rabbit)

Here is my HAProxy setup:

global
    daemon
    maxconn 256

defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

listen http-in
    balance roundrobin
    option forwardfor # This sets X-Forwarded-For
    timeout queue 5000
    timeout server 86400000
    timeout connect 86400000
    bind *:8888
    server server1 18.181.3.164:8888 # ether1

In my nginx configuration, I have inserted:

location ~* \.(eot|ttf|woff)$ {
            add_header Access-Control-Allow-Origin *;
        }

to make sure its not an access control problem (the console does not say it is, so it shouldnt be).

I have also tried adding 

option http-server-close
option http-pretend-keepalive

to my HAProxy config, but to no avail.

Any ideas?

** I am testing in Chrome 9.0.597 and Firefox 3.6 (so with both web sockets, and without, same thing)

Upvotes: 2

Views: 1684

Answers (1)

Willy Tarreau
Willy Tarreau

Reputation: 3424

I don't know for the other components involved in this setup, but last time I checked (a few months ago), nginx did not yet support the Upgrade+101 HTTP mechanism which is used by WebSocket. So maybe your test works until the connection is upgraded ? You should definitely enable logging on haproxy, you'd know where connections are closed and why. BTW, upgrading to 1.4.13 will solve a few logging issues that will help you troubleshooting with more certainty.

Upvotes: 2

Related Questions