CODEWITHSUNDEEP
google-cloud-firestore
honor
honor

Reputation: 8098

google cloud firestore 1 write per second limit for a document and how to get around it

In this video https://youtu.be/o7d5Zeic63s?t=145 it says there is a 1 write per second limit for a document.

If you are making a review application and you want to keep the number of reviews for 5,4,3,2,1 star ratings for a restaurant document how would you do that for an app with a large number of concurrent users then? Think of a popular app like foursquare... I think it would not be possible to keep the number of ratings in the restaurant document itself in this case like so:

Restaurant:
 - name
 - address
 - ratings
  - 1 start: count
  - 2 stars: count
  - 3 starts: count
  - 4 stars: count
  - 5 starts count

This would fail in case of more than 1 update attempts on the number of stars counts for the same restaurant, which is quite possible in this popular app case.

In this case, I can think of keeping a RATINGS sub-collection and recording a rating document for each rating. Then I could get the count of the ratings for a 4-star review. But, when I try to get the count of the documents with 4-star rating would it be like 30k read-billing if there are 30k 4-star ratings? How would you get this count without being charged for 30k reads?

How would you avoid this and how would you keep/update the number of star ratings for a restaurant?

Edit: I have seen this post: How to get a count of number of documents in a collection with Cloud Firestore

None of the proposed solutions work in this case. Since we are assuming there will be more than 1 increment/decrement per second in the count for a star rating.

Upvotes: 1

Views: 1829

Answers (1)

honor
honor

Reputation: 8098

I think I found the solution and I want to share it with you.

To solve this problem we can use Distributed Counters as described here: https://firebase.google.com/docs/firestore/solutions/counters

Write throughput increases linearly with the number of shards, so a distributed counter with 10 shards can handle 10x as many writes as a traditional counter.

To sum up, you create shards for each star rating. For example, we could create 10 shards for each star rating (a total of 50 shards), then we can get the count for 4stars rating by summing up the values inside those 10 shards of 4stars rating.

Keep in mind these limitations:

Shard count - The number of shards controls the performance of the distributed counter. With too few shards, some transactions may have to retry before succeeding, which will slow writes. With too many shards, reads become slower and more expensive.

Cost - The cost of reading a counter value increases linearly with the number of shards, because the entire shards subcollection must be loaded.

Below is the sample code from the firestore documentation I linked above.

Initialize a distributed counter:

function createCounter(ref, num_shards) {
    var batch = db.batch();

    // Initialize the counter document
    batch.set(ref, { num_shards: num_shards });

    // Initialize each shard with count=0
    for (let i = 0; i < num_shards; i++) {
        let shardRef = ref.collection('shards').doc(i.toString());
        batch.set(shardRef, { count: 0 });
    }

    // Commit the write batch
    return batch.commit();
}

Choose a random shard and increase the count:

function incrementCounter(db, ref, num_shards) {
    // Select a shard of the counter at random
    const shard_id = Math.floor(Math.random() * num_shards).toString();
    const shard_ref = ref.collection('shards').doc(shard_id);

    // Update count
    return shard_ref.update("count", firebase.firestore.FieldValue.increment(1));
}

Query for all the shards and sum their count fields:

function getCount(ref) {
    // Sum the count of each shard in the subcollection
    return ref.collection('shards').get().then(snapshot => {
        let total_count = 0;
        snapshot.forEach(doc => {
            total_count += doc.data().count;
        });

        return total_count;
    });
}

Upvotes: 2

Related Questions