CODEWITHSUNDEEP
node.jsmongodbdatabase-cursor
jmdiego
jmdiego

Reputation: 552

MongoDB NodeJS Native cursor closing prematurely

I'm executing a process to each element of a ursor, my collection has around 6 million documents, the entire process would take up to 10 hours since I have to process a profile at a time and it's a somewhat complex process for each one.

 var cursor = dbMain.collection("profiles").find({});

 var getNext = function(){
      cursor.nextObject(processOne);
 };

 var processOne = function(err, profile){
      if(err){
           console.error("Error loading profile", err);
           getNext();
      } else if(profile == null){
           // process is done, all profiles processed!!!
      } else {
           processProfile(profile)
               .then(function(){
                    getNext();
               })
               .catch(function(errProfile){
                    console.error("Error processing profile", errProfile);
                    getNext();
               });
      }
 };

 var processProfile = function(profile){
      // Complex profile processing and promise resolve when done
 };

 getNext();

PROBLEM: I get null (which would mean the cursor has been exhausted) when i'm around 300,000 profiles, so, only the first 300,000 or so get processed. ¿Anyone knows how to deal with it or the reason I'm getting null way before time?

Upvotes: 3

Views: 3988

Answers (3)

sk7796
sk7796

Reputation: 1

I had the same issue, i resolved it by pushing each document of the cursor on a list and then returning the list, as shown here:

const list:any= [];
const cursor= await client.db('insertYourDb').collection('insertYourCollection').find().forEach(function(document) { 
        list.push(document) 
    });
return list;

Upvotes: 0

m.spyratos
m.spyratos

Reputation: 4219

For anyone who stumbles upon this one on latest MongoDB drivers, be aware of the following. If you call .hasNext() twice after the last .next() call, you will get: Error: Cursor is closed.

For example:

while(await cursor.hasNext()) {
    await cursor.next();
}

if (await cursor.hasNext()) {} // Error!

if (!cursor.isClosed() && await cursor.hasNext()) {} // Correct way

Upvotes: 9

jmdiego
jmdiego

Reputation: 552

It seems I've found the reason:

Since it's a cursor that needs to be alive a long time, in the find options I had to add "timeout : false".

 var cursor = dbMain.collection("profiles").find({}, {timeout : false});

In the driver documentation says it's false by default (well it's not!!) , now all of my profiles get processed.

Upvotes: 5

Related Questions