CODEWITHSUNDEEP
firebasegoogle-cloud-platformgoogle-cloud-firestoreretool
Nicholas Sartor
Nicholas Sartor

Reputation: 23

Bulk create new records without overwriting in Firestore

I'm ingesting in Firestore a large number of records (company information) from CSV files. I could be creating 100K+ new records from one single file.

The requirement is that if the record is already there (I use the company domain as ID) it should not be overwritten.

I've initially used Batches as it was the thing I was familiar, but as this doesn't fulfill the requirement of not overwriting, I went on to read about Transactions.

The problem is then execution time. I'm implementing this in a Retool app (Javascript), so timeouts are a thing.

The best thing I could come up is: parallelize some 5K Transaction calls with a Promise all and do several serial batches.

In that page the docs say:

Note: For bulk data entry, use a server client library with parallelized individual writes. Batched writes perform better than serialized writes but not better than parallel writes. You should use a server client library for bulk data operations and not a mobile/web SDK.

Is there any limit to how many parallel calls I can to to Firestore? Can I do more than 5K

Is this the best approach you can think of? Anything that I should take into consideration like ramping up the traffic?

Thanks for any help! 🫨

Upvotes: 0

Views: 52

Answers (1)

Alex Mamo
Alex Mamo

Reputation: 139019

The requirement is that if the record is already there (I use the company domain as ID) it should not be overwritten.

In this case, you should check if the record already exists and only create a new document if it doesn't. Since you're using Firestore, such an operation requires an extra read.

When it comes to transactions, please note that the transactions will fail when the client is offline. So always make sure to be online when you perform transactions.

Upvotes: 2

Related Questions