CODEWITHSUNDEEP
firebasegoogle-cloud-firestore
God Himself
God Himself

Reputation: 219

Work around firestore document size limit?

I need to store a large number of fields, like for a star rating system, but firestore only allows 20,000 fields per document. Is there a known way around this? Right now I am going to 'shard' the fields in multiple documents, and keep the size of each document in a documentSizeTracker document that I use to determine which document to shard to (and add to the counter with a transaction). Is this the correct approach? Any problems with this?

Upvotes: 2

Views: 5158

Answers (2)

aldobaie
aldobaie

Reputation: 1407

Actually, I ran into similar problem with the read and write issues with Firebase. So, here is my conclusion:

# if something small needs to be written & read very often, then use Firebase Realtime Database

  • Firebase Realtime database allows fast writes, but limits concurrent users to 100,000
  • Firebase Firestore allows a maximum of 1 write per second per document
  • It's very expensive to read a document that only contains a rating for example in Firestore

# if something (larger) needs to be read very often with writes usually more than 1 second in between then use Firestore

  • Firestore allows up to 1,000,000 concurrent users at current Beta release (they might make it more)
  • It's cheaper to read a large document (less than 1 MiB limit) in Firestore than Firebase Realtime database

# If your model doesn't fit into these two choices, then you should modify your model and split them into 2 models:

  • 1 very small model to store in Firebase Real Database (ratings for example)
  • 1 larger model to store in Firestore

Note: You could use both Firebase Realtime database and Firebase Firestore in the same project. Don't forget to take into account the billing differences between both databases. and their different limits. I believe, it's best to combine them and use the good side of each instead of trying to force solutions into one of them.

Note 2: I really didn't like the shard-ing idea in Firestore suggested solution and work around

Upvotes: 2

Todd Kerpelman
Todd Kerpelman

Reputation: 17523

Sharding certainly could work. It's hard to say without knowing exactly what kind of data you'll need from your document, and when, but that's certainly a reasonable option. You could also consider having a parent "summary" doc that contains fields you might want to search on and then split all of your data into several documents inside a subcollection of that parent.

One important nuance here: the limit isn't 20,000 fields, but 20,000 indexed fields. So if you're storing a bunch of data inside your document, but you know that you're not going to be searching on all of them, another alternative is to mark some of your fields as unindexed (which you can now do in the Firebase console in the "Exemptions" section).

If you're dealing with thousands of fields, though, you probably won't want to exempt them all one at a time, so a better alternative might be to place your data as a map inside a container field (named something like "allOfMyData"), then just mark that one field as unindexed. That will automatically remove all indexes from any fields contained inside that map.

Upvotes: 3

Related Questions