CODEWITHSUNDEEP
firebasezipgoogle-cloud-storagegoogle-cloud-functions
ktm125
ktm125

Reputation: 452

Cloud Functions: zip multiple documents from Cloud Storage

I already searched through a lot of questions on stack overflow but couldn't find a fitting answer from which I can derive the answer I need:

I want to zip multiple files from a folder within Google Cloud Storage/Firebase Storage with a Cloud Function.

I already found the solution for zipping documents from the local filesystem but could not derive how to do it within a Cloud Function for Cloud Storage.

Upvotes: 4

Views: 5269

Answers (2)

Dabbel
Dabbel

Reputation: 2825

A bit late, but I had the same problem to solve.

The following Firebase Function:

  1. Runs with 1 GB / 120 seconds timeout (for good measure)
  2. Is triggered by WRITE calls (do this only if you have few calls!)
  3. Ignores all paths except background_thumbnail/
  4. Creates a random working directory and deletes it afterwards
  5. Downloads images from Firebase Storage
  6. Zips these images in a folder: background_thumbnail/<IMAGE>
  7. Uploads created ZIP to Firebase Storage
  8. Creates a signed URL for the ZIP file at Firebase Storage
  9. Stores the signed URL in Firestore.

The code can probably be improved and made more elegant, but it works (for now).


const {v4: uuidv4} = require("uuid"); // for random working dir
const JSZip = require("jszip");

exports.generateThumbnailZip = functions
  .runWith({memory: "1GB", timeoutSeconds: 120})
  .region("europe-west3")
  .storage.object()
  .onFinalize(async (object) => {

  // background_thumbnail/ is the watched folder

  if (!object.name.startsWith("background_thumbnail/")) {
    return functions.logger.log(`Aborting, got: ${object.name}.`);
  }

  const jszip = new JSZip();

  const bucket = admin.storage().bucket();
  const fileDir = path.dirname(object.name);
  const workingDir = path.join(os.tmpdir(), uuidv4()); 

  const localZipPath = path.join(workingDir, `${fileDir}.zip`);
  const remoteZipPath = `${fileDir}.zip`;

  await mkdirp(workingDir);

  // -------------------------------------------------------------------
  // DOWNLOAD and ZIP
  // -------------------------------------------------------------------

  const [files] = await bucket.getFiles({prefix: `${fileDir}/`});

  for (let index = 0; index < files.length; index++) {
    const file = files[index];
    const name = path.basename(file.name);
    const tempFileName = path.join(workingDir, name);

    functions.logger.log("Downloading tmp file", tempFileName);

    await file.download({destination: tempFileName});

    jszip.folder(fileDir).file(name, fs.readFileSync(tempFileName));
  }

  const content = await jszip.generateAsync({
    type: "nodebuffer",
    compression: "DEFLATE",
    compressionOptions: { level: 9 }
  });

  functions.logger.log("Saving zip file", localZipPath);

  fs.writeFileSync(localZipPath, content);

  // -------------------------------------------------------------------
  // UPLOAD ZIP
  // -------------------------------------------------------------------

  functions.logger.log("Uploading zip to storage at", remoteZipPath);

  const uploadResponse = await bucket
    .upload(path.resolve(localZipPath), {destination: remoteZipPath});

  // -------------------------------------------------------------------
  // GET SIGNED URL FOR ZIP AND STORE IT IN DB
  // -------------------------------------------------------------------

  functions.logger.log("Getting signed URLs.");

  const signedResult = await uploadResponse[0].getSignedUrl({
    action: "read",
    expires: "03-01-2500",
  });

  const signedUrl = signedResult[0];

  functions.logger.log("Storing signed URL in db", signedUrl);

  // Stores the signed URL under "zips/<WATCHED DIR>.signedUrl"
  await db.collection("zips").doc(fileDir).set({
    signedUrl: signedUrl,
  }, {merge: true});

  // -------------------------------------------------------------------
  // CLEAN UP
  // -------------------------------------------------------------------

  functions.logger.log("Unlinking working dir", workingDir);

  fs.rmSync(workingDir, {recursive: true, force: true});

  functions.logger.log("DONE");

  return null;
});

Upvotes: 2

Mohammad I
Mohammad I

Reputation: 94

Google Cloud Storage supports the decompressive form of transcoding but not a compressive form of transcoding. However, at Cloud Storage, any user can store a gzip-compressed file.

To zip multiple documents from Cloud Storage using Cloud Functions, you can download the files from Cloud Storage to functions instances using gcs.bucket.file(filePath). download, zip the file, and re-upload the files to the Cloud Storage. Here you will find an example of downloading, transforming, and uploading a file. You can find an example to zip multiple files in this StackOverflow thread. This document explains how you can upload objects to Cloud Storage using Console, Gsutil, Code Sample, or REST APIs.

Upvotes: 4

Related Questions