CODEWITHSUNDEEP
mongodbpymongo
extraDarker
extraDarker

Reputation: 119

How to stream data from mongodb?

The mongo find(), return all the documents in a collection. I want to store every document that I have in my mongodb in to another database. So, I want to keep polling mongodb for single new document, because find().limit(1) would keep returning me the same again and again. I just want mongo to return, all the document one by one. Is it possible? How can this be done?

Edit: I want to just copy every entry in my mongodb instance to another db, not just the latest ones.

Upvotes: 0

Views: 7742

Answers (2)

AllHailMegatron
AllHailMegatron

Reputation: 91

It sounds like Philipp's comment has the answer you're looking for, but to answer the original question:

I wouldn't suggest using a sort on the whole database if you expect your document sizes to be quite large, also a limit would result in horrible performance.

Running a find() operation actually returns a cursor object to you, which you can iterate upon, which is effectively streaming and therefore exactly what you're asking for.

Upvotes: 2

DhruvPathak
DhruvPathak

Reputation: 43235

  1. Use sort , by default it is a natural sort i.e.:

    find().sort({_id:-1}).limit(1)

You will need to store the _id of current document retreived and match it with the next one retreived with the same query, if they are different, you store the document, and also the new _id.

  1. Better use tailable cursor : http://docs.mongodb.org/manual/tutorial/create-tailable-cursor/

Upvotes: 3

Related Questions