CODEWITHSUNDEEP
mongodb
kraf
kraf

Reputation: 1309

How do I move a tailable cursor with awaitdata to the end so I just get new updates

I am trying to watch the MongoDB oplog with the node.js driver and it works in theory, but it has quite the ramp up time because it seems to be scanning the entire collection. I found this in the MongoDB docs:

  • Because tailable cursors do not use indexes, the initial scan for the query may be expensive; but, after initially exhausting the cursor, subsequent retrievals of the newly added documents are inexpensive.

Is there a way to quickly "exhaust" the cursor to just start tailing? It seems to me that the Meteor guys have solved this, but I have troubles understanding the difference from reading their code. This is what I currently have:

var cursorOptions = {
    tailable: true,
    awaitdata: true,
    numberOfRetries: -1
};

var oplogStream = oplogDb.collection('oplog.rs').find(
    {
        ns: { $regex : /^dbname\./ },
        op: "i",
        ts: { $gt: lastEntry.ts } 
    },
    cursorOptions
).sort({$natural: -1}).stream();

oplogStream.on('data', publishDocument);

oplogStream.on('end', function() {
    log.error("received unexpected end event from oplog watcher.");
});

Upvotes: 4

Views: 1280

Answers (1)

kraf
kraf

Reputation: 1309

Great, 5 minutes after asking I find the answer. I'll post this here for future reference:

You have to add the oplogReplay flag and set it to true. This only works if you also do a range query on the ts field. I tried this before without having the range set and it did nothing. Above code works when you add this one line highlighted below:

var cursorOptions = {
  tailable: true,
  awaitdata: true,
  oplogReplay: true, // add this line
  numberOfRetries: -1
};

Upvotes: 4

Related Questions