CODEWITHSUNDEEP
architecturecloudgoogle-cloud-data-transfer
mp3por
mp3por

Reputation: 1852

Architecture of app with lots of data transfer

I want to design an app which handles very large amount of data transfer.

The app is very simple - there is front-end, back-end, database and storage. When the user goes on the front-end he can upload an audio file and chose a format to which the file will be converted. The front-end sends the file to the back-end. The back-end checks if it has this file already converted by asking the database. If it has it, the back-end returns the download URL from the database. If it does not have it, it sends the file to a converter function, when the converter is done, it downloads the file from the converter, stores it in the storage and creates an entry in the database. Then it sends the download URL back to the client.

I am now looking at Firebase for facilitating the whole project. I am implementing it with Firebase auth for login/registration, firestore for database, express app in a cloud function for back-end and cloud storage for storage.

My concerns is the pricing and scalability. I expect at least 1000 concurrent users. And I expect a large amount of files to be transferred from/into the storage and the cloud function. The conversion is not handled by this app, so there is basically no computation.

What would be the best architecture for such an app ? Which cloud providers provide lowest prices for data transfer and storage ? How to make my app scalable so that it can handle many users at the same time ?

Upvotes: -2

Views: 41

Answers (1)

Baris Senyerli
Baris Senyerli

Reputation: 920

You can scale your cloud functions with configuration:

...
const { setGlobalOptions } = require("firebase-functions/v2");

setGlobalOptions({
  maxInstances: 10,
  region: "europe-west3",
  timeoutSeconds: 120,
  memory: "512MiB",  // You can increase though
});
...

Your limitation is calculation time and memory bound problem for cloud functions. So you need to divide your files then do it batch by batch. You may write a cloud trigger function after setting up storage saving the downloadable url can be process by a cloud function trigger after saving it to firestore. If you need concurrent users more than 1000 your hot functions count which is maxInstances may you increase as well. But we know that cloud functions have already scaling on calling being hot at function only less latency on first execution.

Upvotes: 1

Related Questions