CODEWITHSUNDEEP
firebasegoogle-cloud-firestore
Frank van Puffelen
Frank van Puffelen

Reputation: 598775

How fast is counting documents in Cloud Firestore?

Last year Firestore introduced count queries, which allows you to retrieve the number of results in a query/collection without actually reading the individual documents.

The documentation for this count feature mentions:

Aggregation queries rely on the existing index configuration that your queries already use, and scale proportionally to the number of index entries scanned. This means that aggregations of small- to medium-sized data sets perform within 20-40 ms, though latency increases with the number of items counted.

And:

If a count() aggregation cannot resolve within 60 seconds, it returns a DEADLINE_EXCEEDED error.

How many documents can Firestore actually count within that 1 minute timeout?

Upvotes: 7

Views: 683

Answers (1)

Frank van Puffelen
Frank van Puffelen

Reputation: 598775

I created some collections with many documents in a test database, and then ran COUNT() queries against that.

The code to generate the minimal documents through the Node.js Admin SDK:

const db = getFirestore();
const col = db.collection("10m");
let count = 0;

const writer = db.bulkWriter();
while (count++ < 10_000_000) {
  if (count % 1000 === 0) await writer.flush();
  writer.create(col.doc(), {
    index: count,
    createdAt: FieldValue.serverTimestamp()
  })
}
await writer.close();

Then I counted them with:

for (const name of ["1k", "10k", "1m", "10m"]) {
  const start = Date.now();
  const result = await getCountFromServer(collection(db, name));
  console.log(`Collection '${name}' contains ${result.data().count} docs (counting took ${Date.now()-start}ms)`);
}

And the results I got were:

count ms
1,000 120
10,000 236
100,000 401
1,000,000 1,814
10,000,000 16,565

I ran some additional tests with limits and conditions, and the results were always in line with the above for the number of results that were counted. So for example, counting 10% of the collection with 10m documents took about 1½ to 2 seconds.

So based on this, you can count up to around 40m documents before you reach the 60 second timeout. Honestly, given that you're charged 1 document read for every up to 1,000 documents counted, you'll probably want to switch over to stored counters well before that.

Upvotes: 8

Related Questions