How to upload a file into Firebase Storage from a callable https cloud function

Question

I have been trying to upload a file to Firebase storage using a callable firebase cloud function. All i am doing is fetching an image from an URL using axios and trying to upload to storage. The problem i am facing is, I don't know how to save the response from axios and upload it to storage.

First , how to save the received file in the temp directory that os.tmpdir() creates. Then how to upload it into storage. Here i am receiving the data as arraybuffer and then converting it to Blob and trying to upload it. Here is my code. I have been missing a major part i think. If there is a better way, please recommend me. Ive been looking through a lot of documentation, and landed up with no clear solution. Please guide. Thanks in advance.

const bucket = admin.storage().bucket();
const path = require('path');
const os = require('os');
const fs = require('fs');
module.exports = functions.https.onCall((data, context) => {
  try {
    return new Promise((resolve, reject) => {
      const {
        imageFiles,
        companyPIN,
        projectId
      } = data;
      const filename = imageFiles[0].replace(/^.*[\\\/]/, '');
      const filePath = `ProjectPlans/${companyPIN}/${projectId}/images/${filename}`; // Path i am trying to upload in FIrebase storage
      const tempFilePath = path.join(os.tmpdir(), filename);
      const metadata = {
        contentType: 'application/image'
      };
      axios
        .get(imageFiles[0], { // URL for the image
          responseType: 'arraybuffer',
          headers: {
            accept: 'application/image'
          }
        })
        .then(response => {
          console.log(response);
          const blobObj = new Blob([response.data], {
            type: 'application/image'
          });
          return blobObj;
        })
        .then(async blobObj => {
          return bucket.upload(blobObj, {
            destination: tempFilePath    // Here i am wrong.. How to set the path of downloaded blob file
          });
        }).then(buffer => {
          resolve({ result: 'success' });
        })
        .catch(ex => {
          console.error(ex);
        });
    });
  } catch (error) {
    // unknown: 500 Internal Server Error
    throw new functions.https.HttpsError('unknown', 'Unknown error occurred. Contact the administrator.');
  }
});

Answer 1

I'd take a slightly different approach and avoid using the local filesystem at all, since its just tmpfs and will cost you memory that your function is using anyway to hold the buffer/blob, so its simpler to just avoid it and write directly from that buffer to GCS using the save method on the GCS file object.

Here's an example. I've simplified out a lot of your setup, and I am using an http function instead of a callable. Likewise, I'm using a public stackoverflow image and not your original urls. In any case, you should be able to use this template to modify back to what you need (e.g. change the prototype and remove the http response and replace it with the return value you need):

const functions = require('firebase-functions');
const axios = require('axios');
const admin = require('firebase-admin');
admin.initializeApp();

exports.doIt = functions.https.onRequest((request, response) => {
    const bucket = admin.storage().bucket();
    const IMAGE_URL = 'https://cdn.sstatic.net/Sites/stackoverflow/company/img/logos/so/so-logo.svg';
    const MIME_TYPE = 'image/svg+xml';
    return axios.get(IMAGE_URL, { // URL for the image
        responseType: 'arraybuffer',
        headers: {
          accept: MIME_TYPE
        }
      }).then(response => {
        console.log(response);  // only to show we got the data for debugging
        const destinationFile = bucket.file('my-stackoverflow-logo.svg');  
        return destinationFile.save(response.data).then(() => {  // note: defaults to resumable upload
          return destinationFile.setMetadata({ contentType: MIME_TYPE });
        });
      }).then(() => { response.send('ok'); })
      .catch((err) => { console.log(err); })
  });

As a commenter noted, in the above example the axios request itself makes an external network access, and you will need to be on the Blaze or Flame plan for that. However, that alone doesn't appear to be your current problem.

Likewise, this also defaults to using a resumable upload, which the documentation does not recommend when you are doing large numbers of small (<10MB files) as there is some overhead.

---

You asked how this might be used to download multiple files. Here is one approach. First, lets assume you have a function that returns a promise that downloads a single file given its filename (I've abridged this from the above but its basically identical except for the change of INPUT_URL to filename -- note that it does not return a final result such as response.send(), and there's sort of an implicit assumption all the files are the same MIME_TYPE):

function downloadOneFile(filename) {
  const bucket = admin.storage().bucket();
  const MIME_TYPE = 'image/svg+xml';
  return axios.get(filename, ...)
    .then(response => {
       const destinationFile = ...
     });
}

Then, you just need to iteratively build a promise chain from the list of files. Lets say they are in imageUrls. Once built, return the entire chain:

let finalPromise = Promise.resolve();
imageUrls.forEach((item) => { finalPromise = finalPromise.then(() => downloadOneFile(item)); });

// if needed, add a final .then() section for the actual function result

return finalPromise.catch((err) => { console.log(err) });

Note that you could also build an array of the promises and pass them to Promise.all() -- that would likely be faster as you would get some parallelism, but I wouldn't recommend that unless you are very sure all of the data will fit inside the memory of your function at once. Even with this approach, you need to make sure the downloads can all complete within your function's timeout.