Haproxy socket.io websocket proxy always falls back to long polling

Question

Hey i have been trying to get Haproxy proxying websocket connections from socket.io

I have read nearly everything i can find on google, and tried countless variations of the haproxy.cfg but no matter what i try socket.io always falls back to long-polling.

Its worth mentioning that the ws connection work perfectly if i route the connection directly to the socket server im using.

So im using the socket.io client and a tornado tornadio2 websocket server. my current haproxy.cfg taken from here haproxy example conf

defaults
mode    http

frontend all
bind 0.0.0.0:80
mode tcp

maxconn 200000
timeout client 86400000
default_backend www_backend

# Any URL beginning with socket.io will be flagged as 'is_websocket'
acl is_websocket path_beg /socket.io
acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket hdr_beg(Host) -i ws

# The connection to use if 'is_websocket' is flagged
use_backend socket_backend_http if is_websocket

tcp-request inspect-delay 500ms
tcp-request content accept if HTTP   

backend www_backend
option httplog
option httpclose
balance roundrobin
option forwardfor
timeout server 30000
timeout connect 4000
server nginx localhost:81 weight 1 maxconn 1024 check

backend socket_backend_http
mode http
option httplog
option http-server-close
option forceclose
no option httpclose
balance roundrobin
option forwardfor 
timeout queue 5000
timeout server 86400000
timeout connect 86400000
timeout check 1s
server socket1 localhost:3012 weight 1 maxconn 1024 check

the websocket request is correctly being routed to the socket_backend_http but the browser console always displays the following error

Unexpected response code: 400

and then socket.io falls back to long-polling after a short time messages appear as expected. I have tested using the latest version of chrome, safari and firefox all with the same results

I have seen so many people say that they have this working its driving me to dispair! I will be eternally grateful for anyone that manages to solve this problem.

again just to clarify haproxy is bound to pot 80, nginx is running on port 81 and the socket server is running on port 3012. If someone feels it would be useful to see the socket server please leave a comment and ill ammend the post with the code

Thanks in advance

EDIT the current haproxy.cfg is actually causing this error to be raised in the tornado server

Traceback (most recent call last):
File "/home/mike/.virtual-envs/watchdog-app/local/lib/python2.7/site-     packages/tornado/ioloop.py", line 399, in _run_callback
callback()
File "/home/mike/.virtual-envs/watchdog-app/local/lib/python2.7/site-packages/tornado/iostream.py", line 304, in wrapper
callback(*args)
File "/home/mike/.virtual-envs/watchdog-app/local/lib/python2.7/site-packages/tornado/httpserver.py", line 250, in _on_headers
self.request_callback(self._request)
File "/home/mike/.virtual-envs/watchdog-app/local/lib/python2.7/site-packages/tornado/web.py", line 1362, in __call__
handler._execute(transforms, *args, **kwargs)
File "/home/mike/.virtual-envs/watchdog-app/local/lib/python2.7/site-packages/tornado/web.py", line 992, in _execute
self._handle_request_exception(e)
File "/home/mike/.virtual-envs/watchdog-app/local/lib/python2.7/site-packages/tornado/web.py", line 1032, in _handle_request_exception
self.send_error(500, exc_info=sys.exc_info())
File "/home/mike/.virtual-envs/watchdog-app/local/lib/python2.7/site-packages/tornado/web.py", line 688, in send_error
self.finish()
File "/home/mike/.virtual-envs/watchdog-app/local/lib/python2.7/site-packages/tornado/web.py", line 669, in finish
self.request.finish()
File "/home/mike/.virtual-envs/watchdog-app/local/lib/python2.7/site-packages/tornado/httpserver.py", line 422, in finish
self.connection.finish()
File "/home/mike/.virtual-envs/watchdog-app/local/lib/python2.7/site-packages/tornado/httpserver.py", line 183, in finish
assert self._request, "Request closed"

UPDATE

Ok so some further developments. I have manage to get this working with a slightly different set up using stunnel. so nginx now re routes http requests to https where its picked up and the ssl connection is terminated by stunnel, which then forwards the request on port 8443 where haproxy picks its up. the socket handshake is made and everything works as expected which is great but id love someone to be able to enlighten me on why this works with ssl and not with http!!!

Answer 1

I know this is an old question but:

Haproxy isn't capable of proxying websockets (as of Jun 2013). The reason for the errors im pretty sure is because it see's it as a normal http connection and times the connection out. You can make the timeout last for ages but that feels dirty.

Your options are:

• Nginx version 1.4 addes support for proxying websockets see http://nginx.org/
• Use Nodejitsu's websocket proxy http://blog.nodejitsu.com/http-proxy-intro (found it leaked memory and fell over when we put > 5k concurrent connections).
• Client side balancing such as using server<n>.x.com
• Using amazon's route53 to do DNS roundrobin (I say amazon because it supports health checking, but normal DNS will do if you don't need HA)
• panic...

One this to keep in mind is that you also need to do sticky sessions or use a centralized session store. SockJS makes this easier than socket.io because it will put a random string in the url for each session so you can simply balancer based on url. You can use a centralized store such as redis for sessions with Socket.io which I found clumsy.

Both Nginx and the nodejitsu option can terminate your SSL for you as well.

Answer 2

Just to make it clear, there is no HTTP content in WebSocket so there is no need for any content-length header.

Also, there is an error in your configuration. The frontend is in mode TCP, so the ACLs will only match when the request comes fast enough. In fact, sometimes it might partially work by pure luck. Please set your frontend to "mode http" to fix this point, and remove your "tcp-inspect" rules which become useless. BTW, your "http-backend" also lacks the HTTP mode.

I guess you have not checked logs, otherwise you would probably have noticed that they're not really exploitable in TCP :-/

Answer 3

I can't speak about Socket.io, but SockJS is tested and works well behind a recent haproxy (for example 1.4.16). See the sample config:

• https://github.com/sockjs/sockjs-node/blob/master/examples/haproxy.cfg

I haven't yet figured out a way to run SockJS behind Nginx.

Answer 4

Did some quick research - not personally familiar, but I've been using SockJS and noticed the same behavior. What I have found at the very least is that Nginx isn't compatible with websockets and HAProxy takes work to get working as you've found. haproxy and socket.io not fully working

The sample configuration, which is very similar to your configuration from the following source http://book.mixu.net/ch13.html, looks 99% similar to your config.

https://github.com/mixu/sioconfig/blob/master/single.haproxy.cfg

The other two tidbits I've found are - from HAProxy + WebSocket Disconnection

• Make sure your HAProxy is 1.4 or better
• Make sure your socket.io is version 0.6.8 or better