CODEWITHSUNDEEP
google-cloud-storagegoogle-api-nodejs-client
Trevor Lazarus
Trevor Lazarus

Reputation: 159

Google Cloud Storage NodeJS and JSON API not returning certain folders when trying to list

While trying to use the NodeJS Client Library (getFiles - https://googleapis.dev/nodejs/storage/latest/Bucket.html#getFiles) and the JSON API (objects/list - https://cloud.google.com/storage/docs/json_api/v1/objects/list) The response seems to not return certain folders in both cases. enter image description here

While using gsutil ls it lists all the folders as expected and using the Google Cloud Console to browse the GCS bucket also lists all files as expected.

Has there been change in the Storage API that could be causing this to happen?

UPDATE:

The code to list GCS Folder paths, that uses the NodeJS Client SDK for Storage:

import * as functions from 'firebase-functions';
import { Storage } from '@google-cloud/storage';
import { globVars } from '../admin/admin';

const projectId = process.env.GCLOUD_PROJECT;
// shared global variables setup
const { keyFilename } = globVars;
// Storage set up
const storage = new Storage({
  projectId,
  keyFilename,
});

export const gcsListPath = functions
  .region('europe-west2')
  .runWith({ timeoutSeconds: 540, memory: '256MB' })
  .https.onCall(async (data, context) => {
    if (context.auth?.token.email_verified) {
      const { bucketName, prefix, pathList = false, fileList = false } = data;
      let list;
      const options = {
        autoPaginate: false,
        delimiter: '',
        prefix,
      };

      if (pathList) {
        options.delimiter = '/';
        const verboseResponse = await storage
          .bucket(bucketName)
          .getFiles(options);
        list = verboseResponse[2].prefixes;
      }
      if (fileList) {
        const [files] = await storage
          .bucket(bucketName)
          .getFiles(options);
        list = files.map((file) => file.name);
      }

      return { list };

    } else {
      return {
        error: { message: 'Bad Request', status: 'INVALID_ARGUMENT' },
      };
    }
  });

Upvotes: 0

Views: 325

Answers (1)

Trevor Lazarus
Trevor Lazarus

Reputation: 159

Looks like the pagination feature was limiting the results. Implementing nextQuery returns the next page of listings.

Upvotes: 1

Related Questions