CODEWITHSUNDEEP
node.jssocketsnginxwebsocketsocket.io
mkornblum
mkornblum

Reputation: 534

Socket.io falling back to polling (websocket gives 400) behind nginx proxy when remote but not local

I'm using nginx 1.4.6 as a proxy to both a Django app and a related NodeJS app that uses socket.io 1.2.1 along with socketio-jwt 2.3.5

Locally, I'm coding using a Vagrant instance (ubuntu 14.04.1), while remotely I'm using a dedicated AWS EC2 instance -- both are configured using the same Vagrantfile and setup shell script so they're very close to identical.

When running locally, my client connects to the nginx server which upgrades the connection and passes it to the socket.io server implementation under Node. Everything works really well.

Remotely, I log a first connection with transport=polling which also instructs the client to upgrade to websocket transport, then a request second with transport=websockets and then a third with transport=polling again.

While the application still works, polling < websockets for many reasons, and I hope I can figure out a way to sort this.

The second request is answered with a 400 Bad Request. Running my Node app with DEBUG=* node index.js gives the following output: 

engine intercepting request for path "/socket.io/" +24s
engine handling "GET" http request "/socket.io/?token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6Im1hcmNlbCIsIm9yaWdfaWF0IjoxNDE4MDU5ODg1LCJ1c2VyX2lkIjozLCJlbWFpbCI6IiIsImV4cCI6MTQxODE0NjI4NX0.2PE0TNRol9G4hby4OzQ-af2e0yjfgFAb-gQJF5tKWRwxWnFLv1NGp3Yo87UaqNaQceMW6KqzMIx2gLcRFnk09A&EIO=3&transport=polling&t=1418062322853-0" +0ms
engine handshaking client "6YDFOjBfvZ9UyutVAAAB" +1ms
engine:socket sending packet "open" ({"sid":"6YDFOjBfvZ9UyutVAAAB","upgrades":["websocket"],"pingInterval":25000,"pingTimeout":60000}) +0ms
engine:polling setting request +0ms
engine:socket flushing buffer to transport +0ms
engine:polling writing "  �0{"sid":"6YDFOjBfvZ9UyutVAAAB","upgrades":["websocket"],"pingInterval":25000,"pingTimeout":60000}" +1ms
engine:socket executing batch send callback +1ms
socket.io:server incoming connection with id 6YDFOjBfvZ9UyutVAAAB +1.7m
socket.io:client connecting to namespace / +1.7m
socket.io:namespace adding socket to nsp / +1.7m
socket.io:socket socket connected - writing packet +1.7m
socket.io:socket joining room 6YDFOjBfvZ9UyutVAAAB +0ms
socket.io:client writing packet {"type":0,"nsp":"/"} +79ms
socket.io-parser encoding packet {"type":0,"nsp":"/"} +1.7m
socket.io-parser encoded {"type":0,"nsp":"/"} as 0 +0ms
engine:socket sending packet "message" (0) +79ms
socket id: "6YDFOjBfvZ9UyutVAAAB" connected to user: "marcel"
socket.io:socket joined room 6YDFOjBfvZ9UyutVAAAB +1ms
engine intercepting request for path "/socket.io/" +121ms
engine handling "GET" http request "/socket.io/?token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6Im1hcmNlbCIsIm9yaWdfaWF0IjoxNDE4MDU5ODg1LCJ1c2VyX2lkIjozLCJlbWFpbCI6IiIsImV4cCI6MTQxODE0NjI4NX0.2PE0TNRol9G4hby4OzQ-af2e0yjfgFAb-gQJF5tKWRwxWnFLv1NGp3Yo87UaqNaQceMW6KqzMIx2gLcRFnk09A&EIO=3&transport=polling&t=1418062323048-1&sid=6YDFOjBfvZ9UyutVAAAB" +0ms
engine setting new request for existing client +1ms
engine:polling setting request +0ms
engine:socket flushing buffer to transport +1ms
engine:polling writing "�40" +38ms
engine:socket executing batch send callback +1ms
engine intercepting request for path "/socket.io/" +0ms
engine handling "GET" http request "/socket.io/?token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6Im1hcmNlbCIsIm9yaWdfaWF0IjoxNDE4MDU5ODg1LCJ1c2VyX2lkIjozLCJlbWFpbCI6IiIsImV4cCI6MTQxODE0NjI4NX0.2PE0TNRol9G4hby4OzQ-af2e0yjfgFAb-gQJF5tKWRwxWnFLv1NGp3Yo87UaqNaQceMW6KqzMIx2gLcRFnk09A&EIO=3&transport=websocket&sid=6YDFOjBfvZ9UyutVAAAB" +1ms
engine bad request: unexpected transport without upgrade +0ms
engine intercepting request for path "/socket.io/" +120ms
engine handling "GET" http request "/socket.io/?token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6Im1hcmNlbCIsIm9yaWdfaWF0IjoxNDE4MDU5ODg1LCJ1c2VyX2lkIjozLCJlbWFpbCI6IiIsImV4cCI6MTQxODE0NjI4NX0.2PE0TNRol9G4hby4OzQ-af2e0yjfgFAb-gQJF5tKWRwxWnFLv1NGp3Yo87UaqNaQceMW6KqzMIx2gLcRFnk09A&EIO=3&transport=polling&t=1418062323180-2&sid=6YDFOjBfvZ9UyutVAAAB" +1ms
engine setting new request for existing client +0ms
engine:polling setting request +0ms

The issue appears to be related to the line engine bad request: unexpected transport without upgrade +0ms, but I don't understand that. The nginx config definitely mentions an upgrade, and it works on my vagrant machine.

The relevant part of the nginx config is as follows:

server {
    ...

    location / {
        ...
    }

    location /socket.io/ {
        proxy_pass http://127.0.0.1:4000/socket.io/;
        proxy_http_version 1.1;

        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-NginX-Proxy true;

        proxy_redirect off;
    }

}

The only point of difference that I can think of between the local and remote setups is that locally, the client connects to localhost on two different ports (I proxy the Node app behind nginx, but the web app runs through Gulp), while remotely the Node app is on a different domain, behind port 80.

Any clues much appreciated :)

Upvotes: 6

Views: 5032

Answers (1)

jk-5
jk-5

Reputation: 51

I have had exactly the same problem. After looking at some tcp captures it looked like nginx did not even receive the incoming Upgrade header, so it didn't pass it on to node.js. (I do suspect cloudflare for doing this, but i am not sure).

The way i fixed this is really hacky, but it does work very well. Basically i modified the nginx configuration to look for the Sec-Websocket-Key header, and when it finds that, it sets the Upgrade header to websocket, and the Connection header to upgrade. This works.

Example configuration:

map $http_sec_websocket_key $upgr {
    ""      "";           # If the Sec-Websocket-Key header is empty, send no upgrade header
    default "websocket";  # If the header is present, set Upgrade to "websocket"
}

map $http_sec_websocket_key $conn {
    ""      $http_connection;  # If no Sec-Websocket-Key header exists, set $conn to the incoming Connection header
    default "upgrade";         # Otherwise, set $conn to upgrade
}

You have to define these 2 map blocks before your server {} block.

Now in your socket.io location add the following 2 lines:

proxy_set_header Upgrade $upgr;
proxy_set_header Connection $conn;

This sets the Upgrade and Connection headers to the mapped values.

I hope this is the solution to your problem as well.

Upvotes: 5

Related Questions