CODEWITHSUNDEEP
firebasefirebase-realtime-databasegoogle-cloud-firestoregoogle-cloud-functions
God Himself
God Himself

Reputation: 219

Complicated data structuring in firebase/firestore

I need an optimal way to store a lot of individual fields in firestore. Here is the problem:

I get json data from some api. it contains a list of users. I need to tell if those users are active, ie have been online in the past n days.

I cannot query each user in the list from the api against firestore, because there could be hundreds of thousands of users in that list, and therefore hundreds of thousands of queries and reads, which is way too expensive.

There is no way to use a list as a map for querying as far as I know in firestore, so that's not an option.

What I initially did was have a cloud function go through and find all the active users maybe once every hour, and place them in firebase realtime database in the structure:

activeUsers{
   uid1: true
   uid2: true
   uid2: true
   etc...
 }

and every time I need to check which users are active, I get all fields under activeUsers (which is constrained to a maximum of 100,000 fields, approx 3~5 mb.

Now i was going to use that as my final mechanism, but I just realised that firebase charges for amount of bandwidth used, not number of reads. Therefore it could get very expensive doing this over and over whenever a user makes this request. And I cannot query every single result from firebase database as, while it does not charge per read (i think), it would be very slow to carry out hundreds of thousands of queries.

Now I have decided to use cloud firestore as my final hope, since it charges for number of reads and writes primarily as opposed to data downloaded and uploaded. I am going to use cloud functions again to check every hour the active users, and I'm going to try to figure out the best way to store that data within a few documents. I was thinking 10,000 fields per document with all the active users, then when a user needs to get the active users, they get all the documents (would be 10 if there are 100,000 total active users) and maps those client side to filter the active users.

So I really have 2 questions. 1, If I do it this way, what is the best way to store that data in firestore, is it the way I suggested? And 2, is there an all around better way to be performing this check of active users against the list returned from the api? Have I got it all wrong?

Upvotes: 1

Views: 167

Answers (2)

Rifat
Rifat

Reputation: 1888

Well this is three years old, but I'll answer here. What you have done is not efficient and not a good approach. What I would do is as follows:

  • Make a separate collection, for all active users.

  • and store all the active users unique field such as ID there.

  • Then query that collection. Update that collection when needed.

Upvotes: 0

Ethan SK
Ethan SK

Reputation: 888

You could use firebase storage to store all the users in a text file, then download that text file every time?

Upvotes: 1

Related Questions