Firestore Concurrent Transactions Freezing

Question

I am having an issue where Firestore transactions freeze when run concurrently. As background, I have a Firestore collection, where each document has a dollar amount and a time. I am creating a function that releases a desired dollar amount from this collection, starting with the oldest documents.

For example, a function call to release $150 will iterate through the collection, removing dollar amounts from the collection until a total of $150 is removed. I do this using a recursive function which 1) finds the oldest dollar amount, 2) removes the inputted amount (i.e. $150) from that number, or deletes the number if the inputted amount is greater than the number, and 3) recurs if there is still a remaining amount to be removed. I use a Firestore transaction for step (2), as there is the possibility this collection will be changed by multiple users at the same time (and note that if I combine (1) and (2) to include the query in the transaction, the code behavior is unchanged).

The code below updates the collection correctly. However, it takes an extremely long time if it is called while an earlier instance is already running: if I call it once, then call it again before the first call is completed, it freezes and has potential to take 20-30 minutes instead of the usual 1-5 seconds. While it is clear that contention between concurrent transactions is causing the freezing (when I remove the write portion of the transaction, there is no freezing), the unknown question is what specifically about the contention is causing the freezing and how to fix it.

Addition: it seems this freezing may be related to https://github.com/firebase/firebase-tools/issues/2452. Consistent with that post, I am facing a 30 second freeze per transaction, which becomes many minutes given a single release has multiple transactions.

 function releaseAmountFromStack(amount) {
  return new Promise((resolve, reject) => {
    let db = admin.firestore();
    let stackRef = db.collection("stack");

    stackRef.orderBy("expirationTime", "asc").limit(1)
      .get().then((querySnapshot) => {
        if(querySnapshot.empty) {
          return reject("None left in stack");
        }

        let itemToRelease = querySnapshot.docs[0];

        releaseItem(itemToRelease.ref, amount)
        .then((actualReleaseAmount) => {
          // If there is still more to release, trigger the next recursion
          // If the full amount has been released, return it
          if (amount > actualReleaseAmount) {
            releaseAmountFromStack(amount-actualReleaseAmount)
            .then((nextActualReleaseAmount) => {
              return resolve(actualReleaseAmount + nextActualReleaseAmount);
            })
            .catch(() => {
              return resolve(actualReleaseAmount);
            });
          } else {
            return resolve(actualReleaseAmount);
          }
        });
    });
  });
}

function releaseItem(itemRef, amountToRelease) {
  let db = admin.firestore();
  return db.runTransaction((transaction) => {
    return transaction.get(itemRef).then((itemDoc) => {
      let itemAmount = itemDoc.data().amount;
      let actualReleaseAmount = Math.min(amountToRelease, itemAmount);

      // If item is exhausted, delete it. Else, update amount
      if (actualReleaseAmount >= itemAmount) {
        transaction.delete(itemDoc.ref);
      } else {
        transaction.set(itemDoc.ref, {
          amount: admin.firestore.FieldValue.increment(-1*Number(actualReleaseAmount)),
        }, {merge: true});
      }
      return actualReleaseAmount;
      });
  });
}

Here are some useful facts from the debug process so far. Thank you so much.

During the freeze, it does not trigger a breakpoint on any of these lines of code. Only when the freeze finishes is a breakpoint triggered. This indicates the delay is not caused by looping through my code (if it were, a breakpoint should be triggered)
The function eventually works as intended in that it releases the correct amount, it just takes a very long time. It will generally freeze, then execute, then freeze, then execute, and so forth until the process is completed
Firestore usage stats show the function executes hundreds of reads and writes, even if it only needs to (and I would expect) it to iterate a few dozen times to release the requisite amount from the collection

Redneys · Accepted Answer

I figured out what caused the elusive freezing. I noticed that the freezing of each transaction was consistently 30 seconds, which prompted me to search for discussion on 30 second transaction freezes. I found (https://github.com/firebase/firebase-tools/issues/2452) which indicates this is an issue with how the Firebase emulator handles concurrent transactions. Indeed, when I deploy the code instead of using the emulator, the freezing is no longer there!

TLDR: the Firebase emulator has an unnatural delay with concurrent transactions.

Firestore Concurrent Transactions Freezing

Answers (2)

Related Questions