CODEWITHSUNDEEP
javascriptnode.jsamazon-web-serviceswebsocket
JzSaiyan
JzSaiyan

Reputation: 23

Node server freezes with lot of websocket data transfer

I'm trying to stream data from a node.js server to lot of clients (> 3000). The node.js server receives the data from one client and broadcast it to all clients. The data is sent once every second and each data is around 2KB. I use ws module to send data via websocket.

The node server is located on an AWS EC2 m3.medium instance which only runs the server.

The clients are simulated on 15 different EC2 t2.micro instances, each launches 300 clients.

Everything is going fine at the beginning but after some time (around 3 min) the server freezes. I can not kill the process or do anything on the server. But if I kill all my clients (by rebooting all the clients instances), the server responds again.

On the server, the CPU is between 40 and 60%, the memory is < 1.5GB (the instance has 4GB), the download bandwidth is around 1.5 Mbits/sec, the upload bandwidth is around 75 Mbits/sec. I did a speedtest and the max bandwidths are around 400 Mbits/sec in download or upload.

Before the server freezes, the memory increases from 300MB to 1000MB. I don't know why and if it is related. I used heapdump and did not manage to understand where the leak was.

Here is the code on the server:

var WebSocketServer = require('ws').Server;
var wss = new WebSocketServer({ port: 8080 });

var users = Array();

wss.binaryType = 'arraybuffer';

wss.on('connection', function(ws) {

  console.log('new user connected');
  users.push(ws);
  console.log(users.length);
  ws.on('error', function(e) {
    console.log('onerror called');
    ws.close()
  });

  ws.on('close', function () {
    console.log('closed');
    users.splice(users.indexOf(ws), 1);
    ws.close()
  });

  ws.on('message', function(data) {
    if (data.length < 10)
      return ;
    var len = users.length;

    for (var i = 0; i < len; i++) {
      users[i].send(data, { binary : true });
    }
  });

});

And here is the code (in python) to send the data:

#!/usr/bin/env python                                                                                                                                                                                       

from time import sleep
from websocket import create_connection

#connection to server                                                                                                                                                                                       
ws = create_connection(server_url)

string = "Mea unum iusto virtute et, et meis munere vix. Meliore sensibus omittantur eum ne, sea quis epicuri sapientem at, fabulas consequat interesset in usu. Vix epicurei platonem ea, in vis agam accu\
sata. Quando maluisset forensibus ut nec, debitis percipitur ad vim, ne vix impetus volutpat. Quo magna viderer ne, nemore doctus copiosae cu mel, id vix dolorem omittam laboramus. Ne populo reprehendunt\
 est, recteque dissentiet delicatissimi vis an. Dolores euripidis complectitur no nam, amet nominavi voluptua ut pri. Vix ex timeam iisque gubergren, ne vim error imperdiet deterruisset. An duo autem vir\
is vituperatoribus. Adversarium instructior te eam. Enim moderatius no eam, ut sit viris populo, ex fugit adolescens inciderint ius. Eum idque dolore voluptatum ex, ex pri solet commune mediocrem. In nib\
h affert pro, mei convenire salutandi argumentum at. Nec in vidisse tamquam. Eos an epicurei suavitate. Ex erat scribentur signiferumque quo. Pro ex sapientem deseruisse. Lorem essent omittam sed ad, pop\
ulo reprehendunt ut sit. Pri maiorum fierent te. Vim aeterno aperiam id. Mea ferri integre eu. Cu per nihil affert, fierent percipit accommodare nam te. Eu qui maiestatis concludaturque, at detracto coti\
dieque vel, no prima essent delicata sea. Nam at appareat reprehendunt. Ubique iudicabit consetetur eu sit. Ius et vivendo propriae prodesset, id his primis platonem, qui nostro quodsi cu. Mea ne wisi mu\
tat facete. Dolorem urbanitas theophrastus ut eam, mei no animal aliquid. Te est movet dicam, id labore latine rationibus his, nullam omnium tincidunt nec ut. Eu mundi ancillae erroribus vis, no vim popu\
lo intellegam. Sonet decore volutpat in has, vidisse appetere reprehendunt vel an, at sea ipsum munere corrumpit. Eos noluisse incorrupte reprehendunt cu, qui quidam intellegebat id, vel audire voluptua \
complectitur ne. Posse iuvaret prodesset vix ea. Urbanitas scriptorem ne eos, te soluta probatus est. Mei ex brute congue, in option saperet mel, veniam ocurreret no quo. Malorum mnesarchum ex quo. Meis \
quaeque pericula duo ei. Ei everti doctus vel."

try:
    for i in range(100000):
        sleep(1)
        ws.send(string)
        print('emit', i)
except:
    ws.close()

ws.close()

As the processor is not at 100% and the bandwidth is less than max bandwidth, do you have any idea of what can happen?

Upvotes: 2

Views: 273

Answers (2)

jcaron
jcaron

Reputation: 17710

As determined in the discussion in the comments, the issue was that the clients were not consuming the data sent to them.

Most likely, this caused the unacknowledged messages to remain in memory waiting for the clients to consume them (even if they did nothing with them), which in turn caused the process to grow and to exceed available memory. It then either started to swap heavily, or exceeded available swap, which is usually not something a system handles very well.

Upvotes: 1

user1555320
user1555320

Reputation: 495

could it be a configuration problem? there is a limit on how many file descriptors can be opened under linux OS. Here there is a good starting point to find how you should setup ou machine to handle as many connection as possible

Upvotes: 0

Related Questions