Reliable way to count total used space of firebase storage per user

Question

I'm trying to figure out an approach that will guarantee correct count of user uploaded file size from a web client to the firebase storage.

The core requirements are these:

• it must be stable: it must guarantee to count every uploaded file
• it must be scalable: I can't read "all stored files" to calculate size at once as the number of files can be huge
• it must be secure: I can't rely on browser, the calculations must be performed on server.

So far the approach is this:

1. user of my web app can upload multiple files
2. those files are uploaded using firebase web client sdk
3. on the server I listen to functions.storage.object().onFinalize() to get file size for each file
4. then I update a dedicated document in firestore to add the new size to the total (let's call it totalStorageSizeDoc)

This approach addresses security and scalability but the problem that I see here is when user is uploading a bunch of small enough files, this can easily trigger lots of functions.storage.object().onFinalize() within 1 second, which is 'not-hard-but-still' a limit for write operations in firestore. At this point the writes to totalStorageSizeDoc will also be performed too fast with the risk of rejecting some of those writes.

Is there a way to easily queue the writes from the onFinalize() to ensure the totalStorageSizeDoc is not overwhelmed? Or should I take a completely different approach? Maybe there are any best-practices out there to count the used storage size that I've missed?

Any advice is much appreciated.

Answer 1

The easiest thing to do would be to enable retries for your function, so that if it fails for whatever reason (like exceeding some limit), then the system will simply retry it until it succeeds.