CODEWITHSUNDEEP
google-cloud-firestore
Jack tileman
Jack tileman

Reputation: 871

Firestore pagination - how to find if there are more data after query (using limit)

We use ndb datastore in our current python 2.7 standard environment. We migrating this application to python 3.7 standard environment with firestore (native mode).

We use pagination on ndb datastore and construct our query using fetch.

 query_results , next_curs, more_flag = query_structure.fetch_page(10)

The next_curs and more_flag are very useful to indicate if there is more data to be fetched after the current query (to fetch 10 elements). We use this to flag the front end for "Next Page" / "Previous Page".

We can't find an equivalent of this in Firestore. Can someone help how to achieve this?

Upvotes: 1

Views: 1734

Answers (2)

Cloudkollektiv
Cloudkollektiv

Reputation: 14729

I build a custom firestore API not long ago to fetch records with pagination. You can take a look at the repository. This is the story of the learning cycle I went through:

My first attempt was to use limit and offset, this seemed to work like a charm, but then I walked into the issue that it ended up being very costly to fetch like 200.000 records. Because when using offset, google charges you also for the reads on all the records before that. The Google Firestore Pricing Page clearly states this:

There are no additional costs for using cursors, page tokens, and limits. In fact, these features can help you save money by reading only the documents that you actually need.

However, when you send a query that includes an offset, you are charged a read for each skipped document. For example, if your query uses an offset of 10, and the query returns 1 document, you are charged for 11 reads. Because of this additional cost, you should use cursors instead of offsets whenever possible.

My second attempt was using a cursor to minimize those reads. I ended up fetching N+1 documents and place the cursor like so:

collection = 'my-collection'
cursor = 'we3adoipjcjweoijfec93r04' # N+1th doc id
q = db.collection(collection)
snapshot = db.collection(collection).document(cursor).get()
q = q.start_at(snapshot) # Place cursor at this document
docs = q.stream()

Google wrote a whole page on pagination in Firestore. Some useful query methods when implementing pagination:

  • limit() limits the query to a fixed set of documents.
  • start_at() includes the cursor document.
  • start_after() starts right after the cursor document.
  • order_by() ensures all documents are ordered by a specific field.

Upvotes: 2

Doug Stevenson
Doug Stevenson

Reputation: 317750

There is no direct equivalent in Firestore pagination. What you can do instead is fetch one more document than the N documents that the page requires, then use the presence of the N+1 document to determine if there is "more". You would omit the N+1 document from the displayed page, then start the next page at that N+1 document.

Upvotes: 5

Related Questions