CODEWITHSUNDEEP
node.jssocket.io
KeJyun
KeJyun

Reputation: 131

NodeJS Socket.io : Many connections in state CLOSE_WAIT and FIN_WAIT2 without release

I used ubuntu(12.04) + nodejs (v0.10.22) + socket.io (v0.9.14) to transmit messages.

There are ~300 simultaneous connections. After some hours (about 1 or 2 hours above, it doesn't show up immediately), some connections will persistent in the state CLOSE_WAIT or FIN_WAIT2.

And these un-dead connections grows linearly with time. The users will hard to connect socket server when the connections number reach the limit (Default 1024) , unless some connections released normally.

The following was socket service connections status, running about 3 hours.

netstat -anl | grep <PORT_OF_NODE_PROCESS> | awk '/^tcp/ {t[$NF]++}END{for(state in t){print state, t[state]} }'

FIN_WAIT2 23
LISTEN 1
CLOSE_WAIT 27
TIME_WAIT 12
ESTABLISHED 333
FIN_WAIT1 12

Probably Solutions

1. Touch js file in regular periods

Using Nodemon Package to run js file, when change the file's last modified time, nodemon will restart service, and release all previous un-dead connections (CLOSEWAIT or FINWAIT2)

2. Increase connections limit

sudo vim /etc/security/limits.conf

*       soft    nofile  1024
*       hard    nofile  2048
root    soft    nofile  4096
root    hard    nofile  8192
user1   soft    nofile  2048
user1   hard    nofile  2048

Try to let connections hard to reach limit.

3. Decrease keep-alive timeout

Let operation system to close connections automatically in the short time, but I'm not try it yet.

Question

I found some probably solution to fix the problem. But the above solutions were not really solved the persistent connections with state CLOSE_WAIT or FIN_WAIT2 problem. I could find this is a result of server(CLOSE_WAIT) or clients (FIN_WAIT2) not correctly closing connections. I think socket.io will force-close these incorrectly connection after some timeout. But it seems like not work correctly.

I try to reappear the state CLOSE_WAIT or FIN_WAIT2 problem in my test environment. But it never show up these connection situation.

  1. After connect socket server and disconnect network
  2. Connect socket server for a long time

I found someone ask related question before (Many stale connections in state CLOSE_WAIT and FIN_WAIT2), but still can't find the solution. Does anyone know how to solve this problem??

Thanks

Upvotes: 9

Views: 8447

Answers (3)

Elad Nava
Elad Nava

Reputation: 7906

If you use the native cluster module and spawn workers, note that if your worker process gets killed forcefully with clients connected to it (i.e. due to low system memory), it will leave behind CLOSE_WAIT sockets which will clog up system resources indefinitely.

The workaround is to kill your main Node.js process as soon as one of your workers gets terminated by the OS. Once the master Node.js process is killed, the system will destroy all sockets in CLOSE_WAIT belonging to dead workers.

Also, it appears that calling socket.destroy() from worker processes on Node.js v4.9.5 also leads to sockets stuck in CLOSE_WAIT state. Updating to Node.js v6.9.5 LTS fixed this for me.

Upvotes: 0

LeoP_TCTO
LeoP_TCTO

Reputation: 1

The above solution may solve CLOSE_WAIT, but does not solve FIN_WAIT2. The latest discussion here(https://github.com/LearnBoost/socket.io/issues/1380) offers possible alternate solutions. This discussion also points out that the problem is in node.js itself and NOT socket.io.

Upvotes: 0

KeJyun
KeJyun

Reputation: 131

I try to use multiple connections to connect socket server at same time, I found that some of the client socket will use the same SOCKET ID(get from xhr and it will looks like nmXTMmCGNQp4EncrfHqj) to establish connection. I close the browser when all connections established, and it will cause many CLOSE_WAIT connections without release. A few of connections will close (Base on number of Unique SOCKET ID that have been generated). Because server will establish TCP/IP connection from SOCKET ID. But, if SOCKET ID connections already exist in connections pool, this connection will not store in connections pool. So when client send FIN packet to try to close connection but not exist in server connections pool. Server will always not send ACK packet to prepare close connection. So these connection will stay in CLOSE_WAIT state and without release.

var host = 'http://socket.server/';
var sockets = [];
for(var i=0;i<200;i++){
    var socket = io.connect(host,{"force new connection":true});
    sockets.push(socket);

  socket.on("message",function(message){
    console.log(message);
  });
  socket.on("disconnect",function(){
    console.log("disconnect");
  });
}

Fix lib\manager.js line 670.

Not to establish TCP/IP connection from SOCKET ID when SOCKET ID connections already exist in connections pool.

See also: https://github.com/kejyun/socket.io/commit/8d6c02a477d365f019530b4ec992420dfb90eb09

if (!this.connected[data.id]) {
  if (transport.open) {
    if (this.closed[data.id] && this.closed[data.id].length) {
      transport.payload(this.closed[data.id]);
      this.closed[data.id] = [];
      }

      this.onOpen(data.id);
      this.store.publish('open', data.id);
      this.transports[data.id] = transport;
    }

    this.onConnect(data.id);
    this.store.publish('connect', data.id);
    //....etc
  }
}

The following was socket service connections status, running about 6 hours.

netstat -anl | grep <PORT_OF_NODE_PROCESS> | awk '/^tcp/ {t[$NF]++}END{for(state in t){print state, t[state]} }'

FIN_WAIT2 37
LISTEN 1
TIME_WAIT 13
ESTABLISHED 295
FIN_WAIT1 20
  1. Benchmarkt socket.io

Upvotes: 4

Related Questions