CODEWITHSUNDEEP
javaperformancemongodb
TestRaj TestLast
TestRaj TestLast

Reputation: 43

Mongo DB Cursor issue while iterating over a huge collection

I am iterating over a Mongo DB collection. This collection has millions of documents in it. I am using java API for this iteration. This iteration exercise can go on for a while and we dont want to impact performance of the application. So we are planning to use Thread.sleep during this iteration. But we are facing cursor timeout kind of problems. Anyone faced this before? Also, is it possible to read data in chunks e.g. 100 documents at a time?

Upvotes: 4

Views: 5642

Answers (1)

Rotem Hermon
Rotem Hermon

Reputation: 2147

Instead of using a cursor over the entire collection you can try paging through the collection by the _id. So each time query for 100 documents (order by _id) and keep the last _id you encounter. Then on each consecutive query use a condition to fetch documents where _id > last _id from previous fetch.

Upvotes: 3

Related Questions