How can I benchmark a websocket-based Node.js application?

Question

I am testing the scalability of a websockets application I wrote using socket.io and Node.js. I want to stress test and benchmark the performance. What are some tools I can use?

So far I've looked into wsbench but I'd like to explore any alternatives.

Answer 1

You can try artillery, artillery supports WebSocket.

https://www.artillery.io/docs/reference/engines/websocket

Answer 2

https://www.npmjs.org/package/thor

It a commandline utility and has similar output as Apache Bench

Thor:                                                  version: 1.0.0

God of Thunder, son of Odin and smasher of WebSockets!

Thou shall:
- Spawn 1 workers.
- Create 2 concurrent/parallel connections.
- Smash 100 connections with the mighty Mjölnir.

The answers you seek shall be yours, once I claim what is mine.

Connecting to ws://server:8080/examples/websocket/chat

  Opened 100 connections


Online               439 milliseconds
Time taken           441 milliseconds
Connected            100
Disconnected         0
Failed               55
Total transferred    132.62kB
Total received       71.85kB

Durations (ms):

                     min     mean     stddev  median max
Handshaking          1       5             3       4 20
Latency              0       0             1       0 3

Percentile (ms):

                      50%     66%     75%     80%     90%     95%     98%     98%    100%
Handshaking          4       5       5       6       7       11      17      20      20
Latency              0       0       1       1       1       1       1       2       3

Answer 3

Spent the day looking into benchmarking node.js and socket.io, I tried a number of solutions, I found you can't use wsbench to benchmark socket.io because socket.io has its own protocol that needs to be addressed to make the connections.

After trying a few other solutions with no luck I ended up using socket.io-benchmark and I just hit 60,000 connections on a m1.large EC2 instance using only one core with room to spare. In my case I only wanted to measure the overhead of the socket.io connections (which required me to comment out some code in socket.io-benchmark) so I didn't have any messages sent between client and server.

It was interesting to see the clients took more CPU and Memory than the server. So make sure the client server you're benchmarking from is beefy, I used a EC2 c1.xlarge and had plenty of head room. And I broke the benchmark into 6 individual processes so it could scale across the multiple cores and avoid memory limits.

Be careful to set the <rampup in seconds> option high enough on the clients, if you set it to low everything buffers and grinds to halt.

The only problem with socket.io-benchmark is I had to use node v0.4.12 to get it all working due to a dependency on v8-profiler which won't compile under newer versions of node (as of writing).

Answer 4

For just benchmarking of how many concurrent connections can be set up I'd recommend wsbench. However it will not work for Socket.IO sites (because more complicated handshake). I was researching it and found a couple of tools that just set up connections to Socket.IO server. It's not enough for measuring real performance. What you need is to send and receive real messages that simulate real client of your web application. I've described some ideas on how to create your own benchmark here: Benchmarking for Node.JS / Socket.IO sites

Answer 5

There's a very simple Socket.IO client implementation for Node.js. It's not an actual benchmarking tool, but it does know about Socket.IO's protocol.

Using that, it's fairly easy to write a client that interacts with your application.