CODEWITHSUNDEEP
javascriptnode.jsmongodb
balafi
balafi

Reputation: 2153

Stream query results with the native mongoDB driver for node

I have 100,000 records in a mongoDB collection and trying to retrieve them in a node.js application using the native driver.

I follow the example in MongoDB doc for CursorStream but get the error: 

RangeError: Maximum call stack size exceeded

Before this error I get many of these:

(node) warning: Recursive process.nextTick detected. This will break in the next version of node. Please use setImmediate for recursive deferral.

here is my code:

var query = {...};
var fields = {...};
var options = {
    // "limit": 10
    //"skip": 10,
    //"sort": title
}

var stream = myCollection.find(query, fields, options).stream();
//  stream.pause();
var results = [];
stream.on('data', function (item){
    results.push(item);
    stream.pause();
    // Restart the stream after 1 miliscecond
    setTimeout(function() {
        stream.resume();
    }, 1);
});

stream.on('close'.....

The error occurs also when I don't define a listener for the 'data' event. but it doesn't occur if a pause the stream right after its creation.

The mongod version is v2.4.1 The node driver version is 1.2.x

Any help/hint would be appreciated.

Upvotes: 2

Views: 2696

Answers (2)

balafi
balafi

Reputation: 2153

It looks like the problem is solved by setting a batch size in the Cursor Stream:

var stream = myCollection.find(query, fields, options).batchSize(10000).stream();

Upvotes: 6

zjonsson
zjonsson

Reputation: 2206

You might want to try setImmediate instead of setTimeout, to ensure outstanding IO operations can be flushed. See also https://github.com/ZJONSSON/streamz/issues/1

Upvotes: 0

Related Questions