CODEWITHSUNDEEP
javascriptgoogle-cloud-functions
Dickson Afful
Dickson Afful

Reputation: 838

Firebase function runs extremely slow

I wrote a firebase function that uses a storage trigger to populate the firestore database for an uploaded csv file. I used and npm package csvtojson to do the conversion from csv to json. The problem is the function runs extremely slow. To be precise it takes a very long time for the data to be reflected in the firestore. There are cases where it doesn't reflect at all. The function is below. Is there a way I can make this faster?

exports.fillDatabaseCSV = functions.storage
  .object()
  .onFinalize(async (object) => {
    try {
      // File and directory paths.
      const filePath = object.name;
      const contentType = object.contentType;
      const tempLocalFile = path.join(os.tmpdir(), filePath);
      const tempLocalDir = path.dirname(tempLocalFile);
      const fileBasename = path.basename(filePath);

      // Exit if this is triggered on a file that is not an image.
      if (contentType !== "text/csv") {
        return console.log("This is not a csv file");
      }

      // Cloud Storage files.
      const bucket = admin.storage().bucket(object.bucket);
      const file = bucket.file(filePath);

      // Create the temp directory where the storage file will be downloaded.
      await mkdirp(tempLocalDir);

      // Download file from bucket.
      await file.download({ destination: tempLocalFile });
      console.log("The csv file has been downloaded to", tempLocalFile);

      //uses csvtojson
      const jsonArray = await csv().fromFile(tempLocalFile);

      let firestorePromise = [];

      jsonArray.forEach(async (item) => {
        let user = await admin.auth().createUser({
          email: item.email,
          password: item.password,
          displayName: item.displayName,
        });

        firestorePromise.push(
          admin.firestore().collection("users").doc(user.toJSON().uid).set({
            email: item.email,
            displayName: item.displayName,
            role: item.role,
          })
        );

        if (item.role !== "normal") {
          let claims = {};
          claims[item.role] = true;
          firestorePromise.push(
            admin.auth().setCustomUserClaims(user.toJSON().uid, claims)
          );
        }
      });

      await Promise.all(firestorePromise);
      //delete temp and file itself
      fs.unlinkSync(tempLocalFile);
      await file.delete();
      return null;
    } catch (error) {
      console.log(error);
    }
  });

Upvotes: 0

Views: 72

Answers (1)

Doug Stevenson
Doug Stevenson

Reputation: 317828

Your functions is dependent on operations from Firestore, Cloud Storage, and Firebase Auth. The products and APIs will always do what they do, and can't be sped up, in a general sense.

Firestore is limited to 500 writes per second per collection "in which documents contain sequential values in an indexed field". You might be coming up against this limit. It's a hard limit and can't be increased. If this is the limiting factor, the best you can do is consider some of the advice in the best practices documentation, including index exemptions for problematic fields in your documents.

Upvotes: 1

Related Questions