Why does my WebSocket connection randomly fail under heavy traffic, even with a load balancer?

Question

I have a WebSocket server running on Node.js with Socket.IO, and everything works fine under low traffic. However, when the number of concurrent WebSocket clients increases (e.g., 10,000+ connections), I start experiencing random disconnections, delayed messages, and sometimes the server crashes.

I am using NGINX as a reverse proxy with the following config:

server {
    listen 80;
    server_name mydomain.com;

    location /socket.io/ {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
    }
}

My Node.js WebSocket server (with Socket.IO):

const io = require("socket.io")(3000, {
    cors: {
        origin: "*",
    },
});

io.on("connection", (socket) => {
    console.log(`User connected: ${socket.id}`);

    socket.on("message", (data) => {
        console.log(`Received: ${data}`);
        socket.emit("response", "Message received!");
    });

    socket.on("disconnect", () => {
        console.log(`User disconnected: ${socket.id}`);
    });
});

1. Clients randomly disconnect (especially under heavy load).
2. Some messages are lost even though WebSockets are meant to be reliable.
3. CPU spikes to 100% when handling thousands of connections.
4. NGINX logs show "Connection reset by peer" errors.

• Increased WebSocket ping intervals → Slight improvement, but disconnections still occur.
• Optimized NGINX configurations for WebSockets → No significant difference.
• Used Redis-based Socket.IO adapter for scaling → Helped a bit but still random disconnections.
• Increased Node.js process memory → Did not prevent high CPU usage under load.
• Tried clustering WebSocket servers → Clients still experience timeouts when reconnecting.

I was expecting my WebSocket server to scale smoothly beyond 10,000 connections without frequent disconnections or high CPU load.