CODEWITHSUNDEEP
iphoneiosxcodeparse-platform
ExpertRK
ExpertRK

Reputation: 51

Unable to fetch more than 10k records

I am developing an app where I have more than 10k records added to a class in parse. Now I am trying to fetch those records using PFQuery( I am using the "skip" property). But I am unable to fetch records beyond 10k and I get the following error message

"Skips larger than 10000 are not allowed"

This is a big problem for me since I need all the data.

Has anybody come across such problem. Please share your views.

Thanks

Upvotes: 3

Views: 732

Answers (2)

Thomas Bouldin
Thomas Bouldin

Reputation: 3725

The problem is indeed due to the cost of mongo skip operations. You can formulate a query such that you don't need the skip operator. My preferred method is to orderBy objectId and then add a condition that objectId > last yielded objectId. This type of query can be indexed and remain fast, unlike skip pagination, which has a O(N^2) cost in seeks.

Upvotes: 3

Taylor Leese
Taylor Leese

Reputation: 52330

My assumption would be that it's based on performance issues with MongoDB's skip implementation.

The cursor.skip() method is often expensive because it requires the server to walk from the beginning of the collection or index to get the offset or skip position before beginning to return result. As offset (e.g. pageNumber above) increases, cursor.skip() will become slower and more CPU intensive. With larger collections, cursor.skip() may become IO bound.

Upvotes: 1

Related Questions