CODEWITHSUNDEEP
node.jsgoogle-cloud-platformgoogle-cloud-storage
Jordan
Jordan

Reputation: 33

How to list all files uploaded to storage bucket in Google Storage Bucket?

I have numerous files uploaded in google cloud storage bucket. I want to perform operations on the names of all files for that particular bucket. how can I achieve it?

Upvotes: 3

Views: 4858

Answers (3)

Jon Catmull
Jon Catmull

Reputation: 12792

Came across this thread and none of the solutions worked for me, I had to use stream to get what I wanted and then wrote my own script and put it in a gist in case it helps others.

https://gist.github.com/JonCatmull/fa55298fd55a565f246ab2e93c5c0e4b

My script just logs all the files over a certain size to a file but you could push them into an array and then perform operations on them if you want.

import { initializeApp } from "firebase-admin/app";
import yargs from "yargs";
import fs from "fs";
import { hideBin } from "yargs/helpers";
import { getStorage } from "firebase-admin/storage";

const argv = yargs(hideBin(process.argv))
  .option("bucket", {
    alias: "b",
    description: "bucket to copy images to",
    type: "string",
    demandOption: true,
  })
  .option("maxSizeMB", {
    alias: "m",
    description: "Max file size in MB",
    type: "number",
    demandOption: true,
  })
  .parse();

// Initialize Firebase
initializeApp({
  storageBucket: argv["bucket"],
});

function logError(msg: string, file = "large-files-error.log") {
  console.error(msg);
  return fs.promises.appendFile(file, `${msg}\n`);
}

function log(msg: string, file = "large-files.log") {
  console.log(msg);
  return fs.promises.appendFile(file, `${msg}\n`);
}

const MB = 1024 * 1024;

try {
  const run = async () => {
    const storage = getStorage();
    const bucket = storage.bucket();

    bucket
      .getFilesStream()
      .on("data", (file) => {
        const size = parseInt(file.metadata.size, 10);
        if (size > argv["maxSizeMB"] * MB) {
          log(`${file.name}
URL: https://storage.googleapis.com/${argv["bucket"]}/${file.name}
Size: ${(size / 1024 / 1024).toFixed(2)}MB
--------------------`);
        }
      })
      .on("error", (e) => {
        console.error(e);
      })
      .on("end", () => {
        console.log("done");
      });
  };
  run();
} catch (e) {
  logError(e);
}

How to setup

Install

Install gcloud https://cloud.google.com/sdk/docs/install

Login and set project

gcloud init
# or just authorize
gcloud auth login
# then set project to the one that contains bucket
gcloud config set project [project-name]

Install script dependencies

npm i ts-node firebase firebase-admin typescript yargs @types/node

Run

Run with ts-node and pass in bucket name and max file size (e.g. 10MB)

ts-node ./index.ts  -b your-bucket.appspot.com -m 10

Logs values to console and also creates a large-files.log file with path, url and size.

Upvotes: 1

Nurul Sundarani
Nurul Sundarani

Reputation: 7638

The below solution is suitable for client side. For Node environment as per question, Refer Doug Stevenson's answer

You need to use listAll() method to get all file Names.

Here is an example from the official documentation

// Create a reference under which you want to list
var listRef = storageRef.child('files/uid');

// Find all the prefixes and items.
listRef.listAll().then(function(res) {
  res.prefixes.forEach(function(folderRef) {
    // All the prefixes under listRef.
    // You may call listAll() recursively on them.
  });
  res.items.forEach(function(itemRef) {
    // All the items under listRef.
  });
}).catch(function(error) {
  // Uh-oh, an error occurred!
});

I would suggest to use list method instead of listAll as the later stores all the results in memory while former uses pagination.

Cloud Storage Documentation

Upvotes: 0

Doug Stevenson
Doug Stevenson

Reputation: 317828

The documentation shows an example of listing all files in a bucket using the provided node SDK. You will want to use the getFiles() method of a Bucket object.

// const bucketName = 'Name of a bucket, e.g. my-bucket';

// Imports the Google Cloud client library
const {Storage} = require('@google-cloud/storage');

// Creates a client
const storage = new Storage();

async function listFiles() {
  // Lists files in the bucket
  const [files] = await storage.bucket(bucketName).getFiles();

  console.log('Files:');
  files.forEach(file => {
    console.log(file.name);
  });
}

listFiles().catch(console.error);

Upvotes: 4

Related Questions