CODEWITHSUNDEEP
node.jsmemory-leaksredisnode-redis
AlexProh
AlexProh

Reputation: 61

Node.js + Redis memory leak. Am I doing something wrong?

var redis = require("redis"),
client = redis.createClient();

for(var i =0 ; i < 1000000; i++){
    client.publish('channel_1', 'hello!');
}

After the code is executed, the Node process consumes 1.2GB of memory and stays there; GC does not reduce allocated memory. If I simulate 2 million messages or 4x500000, node crashes with memory error.

Node: 0.8.*, tried 4.1.1 later but nothing changed
Redis: 2.8 , works well (1MB allocated memory).

My server will be publishing more than 1 million messages per hour. So this is absolutely not acceptable (process crashing every hour).

updated test

var redis = require("redis"),
    client = redis.createClient();

var count = 0;
var x;

function loop(){
    count++;
    console.log(count);

    if(count > 2000){
        console.log('cleared');
        clearInterval(x);
    }
    for(var i =0 ; i < 100000; i++){
        client.set('channel_' + i, 'hello!');
    }

}

x = setInterval(loop, 3000);

This allocate ~ 50Mb, with peak at 200Mb, and now GC drop memory back to 50Mb

Upvotes: 3

Views: 4179

Answers (1)

nick
nick

Reputation: 19794

If you take a look at the node_redis client source, you'll see that every send operation returns a boolean that indicates whether the command queue has passed the high water mark (by default 1000). If you were to log this return value (alternatively, enable redis.debug_mode), there is a good possibility that you'll see false a lot- an indication that you're sending too more requests than Redis can handle all at once.

If this turns out not to be the case, then the command queue is indeed being cleared regularly which means GC is most likely the issue.

Either way, try jfriend00's suggestion. Sending 1M+ async messages with no delay (so basically all at once) is not a good test. The queue needs time to clear and GC needs time to do its thing.

Sources: Backpressure and Unbounded Concurrency & Node-redis client return values

Upvotes: 1

Related Questions