Flutter Firestore pagination in abstract service class

Question

I'm implementing pagination for my Flutter app with Firestore and I am running into a design issue.

I'm using services class to abstract database operation from the business logic of my app through data model class like so:

UI <- business logic (riverpod) <- data model class <- stateless firestore service

This works great as it follows the separation of concerns principles.

However, in the Firestore library, the only way to implement pagination is to save the last DocumentSnapshot to reference it in the next query using startAfterDocument(). This means, as my database services are stateless, I would need to save this DocumentSnapshot in my business logic code, who should in principle be completely abstracted from Firestore.

My first instinct would be to reconstruct a DocumentSnapshot from my data model class inside the service and use that for the pagination, but I would not be able to reconstruct it completely so I wonder if that would be enough.

Has anyone run into this issue? How did you solve it?

Cheers!

Answer 1

I stumbled upon the exact same issue, even though I was using Bloc instead of Riverpod. I wrote a whole article on that, in order to support also live updates to the list and allowing infinite scrolling: ARTICLE ON MEDIUM

My approach was to order the query by name and id (for example), and using startAfter instead of startAfterDocument. For example:

import 'package:cloud_firestore/cloud_firestore.dart';
import 'package:infite_firestore_list/domain/list_item_entity.dart';
import 'package:infite_firestore_list/domain/item_repository.dart';

class FirebaseItemRepository implements ItemRepository {
  final _itemsCollection = FirebaseFirestore.instance.collection('items');

  @override
  Future<Stream<List<ListItem>>> getItems({
    String startAfterName = '',
    String startAfterId = '',
    int paginationSize = 10,
  }) async {
    return _itemsCollection
        .orderBy("name")
        .orderBy(FieldPath.documentId)
        .startAfter([startAfterName, startAfterId])
        .limit(paginationSize)
        .snapshots()
        .map((querySnapshot) => querySnapshot.docs.map((doc) {
              return ListItemDataModel.fromFirestoreDocument(doc).toDomain();
            }).toList());
  }
}

in this way in your logic you only have to use id and name or whatever fields you wish to use, for example a date. If you use a combination of multiple orderBy, the first time you run the query, Firebase may ask you to build the index with a link that will appear in the logs.

The drawback of this approach is that it only works if you are sure that the fields you are using in the orderBy are uniques. In fact, if for example you sort by date, if two fields have the same date and you use startAfter that date (first item), you may skip the second item with the same date...

In my example, the startAfterId doesn't seem useful, but in the usecase I had, it solved some edgecases I stumbled upon.

Alternative

An alternative I thought but that I personally didn't like (hence I did not mention it in my article) could be to store an array of the snapshots of the last documents of each page in the repository itself. Than use the id from the logic domain to request a new page and make the correspondance id <--> snapshot in the repository itself.

This approach could be interesting if you are expecting a finite amount of pages and hence a controlled array in your repository singleton, otherwise it smell memory leaking and that's why I personally do not like this approach to stay as general as possible.

Answer 2

If you are using or can use any orderBy queries. You can use startAfter with your last queries value. For example if you orderBy date you can use last date for your next pagination query.

startAfter method reference

Answer 3

The very definition of paging (you are at one page; you go to the next page) is Stateful, so attempting to do it "stateless" has no meaning.

I don't work in flutter, but in JS/React I built the following class that returns an OBJECT that has the PageForward/PageBack methods, and properties to hold the required data/state:

export class PaginateFetch {
  /**
   * constructs an object to paginate through large Firestore Tables
   * @param {string} table a properly formatted string representing the requested collection
   * - always an ODD number of elements
   * @param {array} filterArray an (optional) 3xn array of filter(i.e. "where") conditions
   * The array is assumed to be sorted in the correct order -
   * i.e. filterArray[0] is added first; filterArray[length-1] last
   * returns data as an array of objects (not dissimilar to Redux State objects)
   * with both the documentID and documentReference added as fields.
   * @param {array} sortArray a 2xn array of sort (i.e. "orderBy") conditions
   * @param {?string} refPath (optional) allows "table" parameter to reference a sub-collection
   * of an existing document reference (I use a LOT of structured collections)
   * @param {number} limit page size
   * @category Paginator
   */
  constructor(
    table,
    filterArray = null,
    sortArray = null,
    refPath = null,
    limit = PAGINATE_DEFAULT
  ) {
    const db = dbReference(refPath);

    /**
     * current limit of query results
     * @type {number}
     */
    this.limit = limit;
    /**
     * underlying query for fetch
     * @private
     * @type {Query}
     */
    this.Query = sortQuery(
      filterQuery(db.collection(table), filterArray),
      sortArray
    );
    /**
     * current status of pagination
     * @type {PagingStatus}
     * -1 pending; 0 uninitialized; 1 updated;
     */
    this.status = PAGINATE_INIT;
  }

  /**
   * executes the query again to fetch the next set of records
   * @async
   * @method
   * @returns {Promise<RecordArray>} returns an array of record - the next page
   */
  PageForward() {
    const runQuery = this.snapshot
      ? this.Query.startAfter(last(this.snapshot.docs))
      : this.Query;

    this.status = PAGINATE_PENDING;

    return runQuery
      .limit(this.limit)
      .get()
      .then((QuerySnapshot) => {
        this.status = PAGINATE_UPDATED;
        //*IF* documents (i.e. haven't gone beyond start)
        if (!QuerySnapshot.empty) {
          //then update document set, and execute callback
          //return Promise.resolve(QuerySnapshot);
          this.snapshot = QuerySnapshot;
        }
        return Promise.resolve(RecordsFromSnapshot(this.snapshot));
      });
  }

  /**
   * executes the query again to fetch the previous set of records
   * @async
   * @method
   * @returns {Promise<RecordArray>} returns an array of record - the next page
   */
  PageBack() {
    const runQuery = this.snapshot
      ? this.Query.endBefore(this.snapshot.docs[0])
      : this.Query;

    this.status = PAGINATE_PENDING;

    return runQuery
      .limitToLast(this.limit)
      .get()
      .then((QuerySnapshot) => {
        this.status = PAGINATE_UPDATED;
        //*IF* documents (i.e. haven't gone back ebfore start)
        if (!QuerySnapshot.empty) {
          //then update document set, and execute callback
          this.snapshot = QuerySnapshot;
        }
        return Promise.resolve(RecordsFromSnapshot(this.snapshot));
      });
  }
}
/**
 * @private
 * @typedef {Object} filterObject
 * @property {!String} fieldRef
 * @property {!String} opStr
 * @property {any} value
 */

/**
 * ----------------------------------------------------------------------
 * @private
 * @function filterQuery
 * builds and returns a query built from an array of filter (i.e. "where")
 * conditions
 * @param {Query} query collectionReference or Query to build filter upong
 * @param {?filterObject} [filterArray] an (optional) 3xn array of filter(i.e. "where") conditions
 * @returns {Query} Firestore Query object
 */
const filterQuery = (query, filterArray = null) => {
  return filterArray
    ? filterArray.reduce((accQuery, filter) => {
        return accQuery.where(filter.fieldRef, filter.opStr, filter.value);
      }, query)
    : query;
};

/**
 * @private
 * @typedef {Object} sortObject
 * @property {!String} fieldRef
 * @property {!String} dirStr
 */

/**
 * ----------------------------------------------------------------------
 * @private
 * @function sortQuery
 * builds and returns a query built from an array of filter (i.e. "where")
 * conditions
 * @param {Query} query collectionReference or Query to build filter upong
 * @param {?sortObject} [sortArray] an (optional) 2xn array of sort (i.e. "orderBy") conditions
 * @returns Firestore Query object
 */
const sortQuery = (query, sortArray = null) => {
  return sortArray
    ? sortArray.reduce((accQuery, sortEntry) => {
        return accQuery.orderBy(sortEntry.fieldRef, sortEntry.dirStr || "asc");
        //note "||" - if dirStr is not present(i.e. falsy) default to "asc"
      }, query)
    : query;
};