Make sure firestore collection docChanges keeps alive

Question

The final solution is at the bottom of this post.

I have a nodeJS server application that listens to a rather big collection:

//here was old code

This works perfectly fine: these are lots of documents and the server can serve them from cache instead of database, which saves me tons of document reads (and is a lot faster).

I want to make sure, this collection is staying alive forever, this means reconnecting if a change is not coming trough.

Is there any way to create this certainty? This server might be online for years.

Final solution:

• database listener that saves the timestamp on a change

export const lastRolesChange = functions.firestore
  .document(`${COLLECTIONS.ROLES}/{id}`)
  .onWrite(async (_change, context) => {
    return firebase()
      .admin.firestore()
      .collection('syncstatus')
      .doc(COLLECTIONS.ROLES)
      .set({
        lastModified: context.timestamp,
        docId: context.params.id
      });
  });

• logic that checks if the server has the same updated timesta.mp as the database. If it is still listening, it should have, otherwise refresh listener because it might have stalled.

import { firebase } from '../google/auth';
import { COLLECTIONS } from '../../../configs/collections.enum';

class DataObjectTemplate {
  constructor() {
    for (const key in COLLECTIONS) {
      if (key) {
        this[COLLECTIONS[key]] = [] as { id: string; data: any }[];
      }
    }
  }
}

const dataObject = new DataObjectTemplate();

const timestamps: {
  [key in COLLECTIONS]?: Date;
} = {};

let unsubscribe: Function;

export const getCachedData = async (type: COLLECTIONS) => {
  return firebase()
    .admin.firestore()
    .collection(COLLECTIONS.SYNCSTATUS)
    .doc(type)
    .get()
    .then(async snap => {
      const lastUpdate = snap.data();

      /* we compare the last update of the roles collection with the last update we
       * got from the listener. If the listener would have failed to sync, we
       * will find out here and reset the listener.
       */

      // first check if we already have a timestamp, otherwise, we set it in the past.
      let timestamp = timestamps[type];
      if (!timestamp) {
        timestamp = new Date(2020, 0, 1);
      }

      // if we don't have a last update for some reason, there is something wrong
      if (!lastUpdate) {
        throw new Error('Missing sync data for ' + type);
      }

      const lastModified = new Date(lastUpdate.lastModified);

      if (lastModified.getTime() > timestamp.getTime()) {
        console.warn('Out of sync: refresh!');
        console.warn('Resetting listener');
        if (unsubscribe) {
          unsubscribe();
        }
        await startCache(type);
        return dataObject[type] as { id: string; data: any }[];
      }
      return dataObject[type] as { id: string; data: any }[];
    });
};

export const startCache = async (type: COLLECTIONS) => {
  // tslint:disable-next-line:no-console
  console.warn('Building ' + type + ' cache.');
  const timeStamps: number[] = [];
  // start with clean array

  dataObject[type] = [];

  return new Promise(resolve => {
    unsubscribe = firebase()
      .admin.firestore()
      .collection(type)
      .onSnapshot(querySnapshot => {
        querySnapshot.docChanges().map(change => {
          timeStamps.push(change.doc.updateTime.toMillis());

          if (change.oldIndex !== -1) {
            dataObject[type].splice(change.oldIndex, 1);
          }
          if (change.newIndex !== -1) {
            dataObject[type].splice(change.newIndex, 0, {
              id: change.doc.id,
              data: change.doc.data()
            });
          }
        });
        // tslint:disable-next-line:no-console
        console.log(dataObject[type].length + ' ' + type + ' in cache.');
        timestamps[type] = new Date(Math.max(...timeStamps));
        resolve(true);
      });
  });
};

Answer 1

If you want to be sure you have all changes, you'll have to:

1. keep a lastModified type field in each document,
2. use a query to get documents that we modified since you last looked,
3. store the last time you queried on your server.

Unrelated to that, you might also be interested in the recently launched ability to serve bundled Firestore content as it's another way to reduce the number of charged reads you have to do against the Firestore server.