CODEWITHSUNDEEP
node.jsgoogle-cloud-platformgoogle-cloud-storage
Mihov
Mihov

Reputation: 5

Read JSON file directly from google storage (using Cloud Functions)

I created a function that extracts a specific attribute from a JSON file, but this file was together with the function in Cloud Functions. In this case, I was simply attaching the file and was able to refer to a specific attribute:

const jsonData = require('./data.json');
const result = jsonData.responses[0].fullTextAnnotation.text;

return result;

Ultimately, I want to read this file directly from cloud storage and here I have tried several solutions, but without success. How can I read a JSON file directly from google storage so that, as in the first case, I can read its attributes correctly?

Upvotes: 0

Views: 2062

Answers (3)

Andi F.
Andi F.

Reputation: 776

If you like to use the JSON from your storage bucket w/o writing it to a temp directory, try the following:

import { getStorage } from "firebase-admin/storage";

async function storageFileToJSON(bucketName: string, fileName: string): Promise<any> {
  const fileContents: any = await getStorage().bucket(bucketName).file(fileName).download();
  return JSON.parse(fileContents.toString('utf8'));
}

Upvotes: 0

guillaume blaquiere
guillaume blaquiere

Reputation: 75715

To clearly answer the question: you can't!

You need to download the file locally first, and then process it. You can't read it directly from GCS.

With Cloud Functions you can only store file in the /tmp directory, it's the only one writable. In addition, it's an in-memory file system, that means several things:

  • The size is limited by the memory set up to the Cloud Function. The memory space is shared between your app memory footprint and your file storage in /tmp (you won't be able to download a file of 10Gb for example)
  • The memory is lost when the instance goes down and
  • All the Cloud Functions instances have their own memory space. You can't share the files between all the Cloud Functions
  • The /tmp directory isn't cleaned between 2 functions invocation (on the same instance). Think to cleanup yourselves this directory.

Upvotes: 0

Gray_Rhino
Gray_Rhino

Reputation: 1283

As mentioned in the comment the Cloud Storage API allows you to do many things through API. Here's an example from documentation on how to download a file from Cloud Storage for your reference.

/**
 * TODO(developer): Uncomment the following lines before running the sample.
 */
// The ID of your GCS bucket
// const bucketName = 'your-unique-bucket-name';

// The ID of your GCS file
// const fileName = 'your-file-name';

// The path to which the file should be downloaded
// const destFileName = '/local/path/to/file.txt';

// Imports the Google Cloud client library
const {Storage} = require('@google-cloud/storage');

// Creates a client
const storage = new Storage();

async function downloadFile() {
  const options = {
    destination: destFileName,
  };

  // Downloads the file
  await storage.bucket(bucketName).file(fileName).download(options);

  console.log(
    `gs://${bucketName}/${fileName} downloaded to ${destFileName}.`
  );
}

downloadFile().catch(console.error);

Upvotes: 1

Related Questions