CODEWITHSUNDEEP
google-cloud-firestoregoogle-bigquerygoogle-cloud-functionsgoogle-cloud-bigtable
Akhil
Akhil

Reputation: 646

Exposing Data from BigQuery to Mobile/Web Apps Via Firestore

I am looking for an easy way (of course with good performance) to expose data in my BigQuery table to web applications.

The current solution which is running is using a Cloud Function and Firestore (in native mode) to expose the data in BigQuery. The implementation is like - as soon as the data is written to the final big query table, we are triggering cloud functions (500 records per commit) to update the data in our final firestore table. The data in firestore table is finally exposed to the App/Web client.

And, to avoid timeout issues associated with Cloud Functions, we are dividing the entire dataset into batches and each cloud function instance will handle a single batch of records only.

But soon after going live, we were hit with scalability issues for the writes as we were triggering the Cloud Function instances sequentially.

A simple way to improve the performance could be to do parallel writes from inside the cloud function, but again as per the firestore documentation doing more than 1000 writes/sec against a collection can reduce performance. So eventually the performance gains we are getting with this approach could be minimum. In our case, we have only one collection.

Anyone here has experience of dealing with high volume writes and reads against Firestore ? Firestore in datastore mode can be used for high volume writes, but what about the read latency?

Also, I am thinking of using BigTable for this purpose (eventual consistency could be fine for us), but using bigtable might add additional layers to expose the data, maybe through a web service.

We are expecting data size to be around GBs only.

PS : I don't need the offline capabilities offered by the Firestore, the reason for choosing Firestore was for the ease of development only. 

Upvotes: 0

Views: 702

Answers (1)

Rafael Lemos
Rafael Lemos

Reputation: 5829

Based on the information you shared Firestore does not seem like an appropriate choice of product for the amount of data you will be adding at once, plus the costs of this might be heavier than the alternative if we talking about TBs of data, which I assume is the case.

Generally speaking Firestore is not recommended for very data intensive apps nor apps with too many writes, for pricing reasons, as reads are considerably cheaper than writes.

Personally I would choose Big Table for this case for the following reasons:

  • Supports apps with high throughput.
  • Easily scalable without lost in performance or instance downtime while doing so.
  • If kept in the same zone or region as Big Query will have no additional costs to migrating the data to Big Table.

Upvotes: 0

Related Questions