Firestore pagination - Is there any query compatible with firebase's limitToLast?

Question

Is there a way to implement back pagination with firestore? I am struggling to implement pagination with firestore, and there are limited firestore queries for it. Forward pagination can be made by startAt and limit method, that is ok. But back pagination can't be easily done, because we only have endBefore, and endAt method, and how can we get last n elements from given document? I know realtime database have method limitToLast. Is there any query like this for firestore? (Also I need to implement multiple sorting, so getting last documents with "ASC" or "DESC" sorting will not work) Help much appreciated.

Thanks!

Answer 1

I just want to share my code for Firestore pagination.
I am using react hooks w/ NextJS.

You will need to have "useFirestoreQuery" hook, which can be found here.
https://usehooks.com/useFirestoreQuery/

So here is my set up.

/* Context User */
const {user} = useUser()

/* States */
const [query, setQuery] = useState(null)
const [ref, setRef] = useState(null)
const [reverse, setReverse] = useState(false)
const [limit, setLimit] = useState(2)
const [lastID, setLastID] = useState(null)
const [firstID, setFirstID] = useState(null)
const [page, setPage] = useState(1)

/* Query Hook */
const fireCollection = useFirestoreQuery(query)

/* Set Ref, **When firebase initialized** */
useEffect(() => {
  user?.uid &&
    setRef(
      firebase
        .firestore()
        .collection('products')
        .where('type', '==', 'vaporizers')
    )
}, [user])

/* Initial Query, **When ref set** */
useEffect(() => {
  ref && setQuery(ref.orderBy('id', 'asc').limit(limit))
}, [ref])

/* Next Page */
const nextPage = useCallback(() => {
  setPage((p) => parseInt(p) + 1)
  setReverse(false)
  setQuery(ref.orderBy('id', 'asc').startAfter(lastID).limit(limit))
}, [lastID, limit])

/* Prev Page */
const prevPage = useCallback(() => {
  setPage((p) => parseInt(p) - 1)
  setReverse(true)
  setQuery(ref.orderBy('id', 'desc').startAfter(firstID).limit(limit))
}, [firstID, limit])

/* Product List */
const ProductList = ({fireCollection}) => {
  const [products, setProducts] = useState([])

  useEffect(() => {
    let tempProducts = []
    let tempIDs = []
    const {data} = fireCollection
    for (const key in data) {
      const product = data[key]
      tempIDs.push(product.id)
      tempProducts.push(<ProductRow {...{product}} key={key} />)
    }
    if (reverse) {
      tempProducts.reverse()
      tempIDs.reverse()
    }
    setFirstID(tempIDs[0])
    setLastID(tempIDs.pop())
    setProducts(tempProducts)
  }, [fireCollection])

  return products
}

I moved the 'ProductList' outside of the component with a context provider, but this is the gist of it.

Note. If you are looking for the total number of products. I suggest you keep up with the totals with these cloud functions. You will need to store your totals in a separate collection. I call mine 'shortcuts'.

exports.incrementProducts = functions.firestore
  .document('products/{id}')
  .onCreate(async (snap, context) => {
    const createdProduct = snap.data()
    /* Increment a shortcut collection that holds the totals to your products */
  })

exports.decrementProducts = functions.firestore
  .document('products/{id}')
  .onDelete((snap, context) => {
    const deletedProduct = snap.data()
    /* Decrement a shortcut collection that holds the totals to your products */
  })

Don't Forget
Make sure you set your indexes for all this to work. Here is what mine looks like.

Answer 2

Simpler answer: Firestore now has .limitToLast(), which works exactly as you think it does. Used in my own (guess I need to publish it soon) Firestore Wrapper:

//////////////////////////////////////////////////////////////////////
//////////////////////////////////////////////////////////////////////
// *** Paginate API ***

export const PAGINATE_INIT = 0;
export const PAGINATE_PENDING = -1;
export const PAGINATE_UPDATED = 1;
export const PAGINATE_DEFAULT = 10;
export const PAGINATE_CHOICES = [10, 25, 50, 100, 250, 500];

/**
 * @classdesc
 * An object to allow for paginating a table read from Firestore. REQUIRES a sorting choice
 * @property {Query} Query that forms basis for the table read
 * @property {number} limit page size
 * @property {QuerySnapshot} snapshot last successful snapshot/page fetched
 * @property {enum} status status of pagination object
 * @method PageForward pages the fetch forward
 * @method PageBack pages the fetch backward
 */

export class PaginateFetch {
  Query = null;
  limit = PAGINATE_DEFAULT;
  snapshot = null;
  status = null; // -1 pending; 0 uninitialize; 1 updated;
  /**
   * ----------------------------------------------------------------------
   * @constructs PaginateFetch constructs an object to paginate through large
   * Firestore Tables
   * @param {string} table a properly formatted string representing the requested collection
   * - always an ODD number of elements
   * @param {array} filterArray an (optional) 3xn array of filter(i.e. "where") conditions
   * @param {array} sortArray a 2xn array of sort (i.e. "orderBy") conditions
   * @param {ref} ref (optional) allows "table" parameter to reference a sub-collection
   * of an existing document reference (I use a LOT of structered collections)
   *
   * The array is assumed to be sorted in the correct order -
   * i.e. filterArray[0] is added first; filterArray[length-1] last
   * returns data as an array of objects (not dissimilar to Redux State objects)
   * with both the documentID and documentReference added as fields.
   * @param {number} limit (optional)
   * @returns {PaginateFetchObject}
   **********************************************************************/

  constructor(
    table,
    filterArray = null,
    sortArray = null,
    ref = null,
    limit = PAGINATE_DEFAULT
  ) {
    const db = ref ? ref : fdb;

    this.limit = limit;
    this.Query = sortQuery(
      filterQuery(db.collection(table), filterArray),
      sortArray
    );
    this.status = PAGINATE_INIT;
  }

  /**
   * @method Page
   * @returns Promise of a QuerySnapshot
   */
  PageForward = () => {
    const runQuery = this.snapshot
      ? this.Query.startAfter(_.last(this.snapshot.docs))
      : this.Query;

    this.status = PAGINATE_PENDING;

    return runQuery
      .limit(this.limit)
      .get()
      .then((QuerySnapshot) => {
        this.status = PAGINATE_UPDATED;
        //*IF* documents (i.e. haven't gone beyond start)
        if (!QuerySnapshot.empty) {
          //then update document set, and execute callback
          //return Promise.resolve(QuerySnapshot);
          this.snapshot = QuerySnapshot;
        }
        return this.snapshot.docs.map((doc) => {
          return {
            ...doc.data(),
            Id: doc.id,
            ref: doc.ref
          };
        });
      });
  };

  PageBack = () => {
    const runQuery = this.snapshot
      ? this.Query.endBefore(this.snapshot.docs[0])
      : this.Query;

    this.status = PAGINATE_PENDING;

    return runQuery
      .limitToLast(this.limit)
      .get()
      .then((QuerySnapshot) => {
        this.status = PAGINATE_UPDATED;
        //*IF* documents (i.e. haven't gone back ebfore start)
        if (!QuerySnapshot.empty) {
          //then update document set, and execute callback
          this.snapshot = QuerySnapshot;
        }
        return this.snapshot.docs.map((doc) => {
          return {
            ...doc.data(),
            Id: doc.id,
            ref: doc.ref
          };
        });
      });
  };
}

/**
 * ----------------------------------------------------------------------
 * @function filterQuery
 * builds and returns a query built from an array of filter (i.e. "where")
 * consitions
 * @param {Query} query collectionReference or Query to build filter upong
 * @param {array} filterArray an (optional) 3xn array of filter(i.e. "where") conditions
 * @returns Firestor Query object
 */
export const filterQuery = (query, filterArray = null) => {
  return filterArray
    ? filterArray.reduce((accQuery, filter) => {
        return accQuery.where(filter.fieldRef, filter.opStr, filter.value);
      }, query)
    : query;
};

/**
 * ----------------------------------------------------------------------
 * @function sortQuery
 * builds and returns a query built from an array of filter (i.e. "where")
 * consitions
 * @param {Query} query collectionReference or Query to build filter upong
 * @param {array} sortArray an (optional) 2xn array of sort (i.e. "orderBy") conditions
 * @returns Firestor Query object
 */
export const sortQuery = (query, sortArray = null) => {
  return sortArray
    ? sortArray.reduce((accQuery, sortEntry) => {
        return accQuery.orderBy(sortEntry.fieldRef, sortEntry.dirStr || "asc");
        //note "||" - if dirStr is not present(i.e. falsy) default to "asc"
      }, query)
    : query;
};

I also have the equivalent for CollectionGroup queries, and listeners for each as well.

Answer 3

Yes. Building upon Frank's answer...

Have something like this in your query...

    if (this.next) {
      // if next, orderBy field descending, start after last field
      q.orderBy('field', 'desc');
      q.startAfter(this.marker);
    } else if (this.prev) {
      // if prev, orderBy field ascending, start after first field
      q.orderBy('field', 'asc');
      q.startAfter(this.marker);
    } else {
      // otherwise just display first page results normally
      q.orderBy('field', 'desc');
    }
    q.limit(this.pageSize);

and then reverse it when you get the query...

    this.testsCollection
            .valueChanges({ idField: 'id' })
            .pipe(
              tap(results => {
                if (this.prev) {
                  // if previous, need to reverse the results...
                  results.reverse();
                }
              })
            )

Answer 4

I was running into this same issue, and not understanding why using limit with endAt wasn't returning the results I desired. I was attempting to implement a list in which you could paginate in both directions, first forward and then backward back to the start of the list.

To remedy the situation I decided to just cache the startAfter DocumentSnapshot for each page so that one can move both directions, in this way I never have to use endAt. The only time this will become an issue is if the collection of documents shifts or changes while the user is on a page other than the first page, but by returning to the first page it will reset to the beginning of the collection.

Answer 5

The equivalent to the limitToLast(...) operation from the Firebase Realtime Database in Cloud Firestore is to order the data descending (which is possible in Firestore) and then just limit(...). If you're having problems implement this, update your question to show what you've done.

I agree that this is a sub-optimal API for back-pagination, since you're receiving the items in reverse order.