CODEWITHSUNDEEP
javascriptnode.jsfirebaseasync-awaitgoogle-cloud-functions
cbdeveloper
cbdeveloper

Reputation: 31365

Trying to do fs.writefile() inside an async cloud function (node v8)

I'm currently building a cloud function to scrape some data from ecommerce websites. I'm running them on the Node.js v8 environment, so I'm able to use async/await without transpiling.

I need a way to save a file with the html response from the website so I can inspect it and decide how to best scrape it. And I haven't been able to make it work.

As I found resarching in many places, I need a way to promisify the fs.writeFile() function so that I can await for it. I've tried with util.promisify and also building the custom function below (both produce the same result, which is my cloud function crashing when I await for the result):

index.js (main file for the cloud functions - simplified version)

const functions = require('firebase-functions');
const request = require('request');
const cheerio = require('cheerio');
const fs = require('fs');

// PROMISIFY WRITEFILE TO SAVE URL RESPONSE
const promisifiedWriteFile = function(text) {
  return new Promise((resolve, reject) => {
    fs.writeFile('./response.html', text, err => {
      if (err) {
        return reject(err);
      }
      return resolve();
    });
  });
};

exports.getUrlProductDetails = functions.https.onCall(
  async (data) => {

    try {
      // SEND REQUEST TO URL PRODUCT PAGE, PARSE WITH CHEERIO
      response = await promisifiedRequest(productURL);
      $ = cheerio.load(response.body);

      // TRYING TO SAVE FILE
      await promisifiedWriteFile('12345'); // <----- THIS IS CRASHING
    } // TRY BLOCK - OUTER - END

    catch (err) {
      console.log(err);
      throw new functions.https.HttpsError('unknown', 'Some internal error', err);
    }

  console.log('End of async function...');

  return {
    productDetails: productDetails,
    body: response.body // My client code is getting the body response just fine
  };

Note1: This is a simplified version of the code, but the single line that is crashing the function is: await promisifiedWriteFile('12345'); and when I comment it out, everything else works just fine.

Note2: The file response.html gets created (with empty content), even with the function crashing at that point.

Note3: I should be trying to write the response.body. The '12345' is just for testing. But it crashes either way.

What am I doing wrong?

EDIT

// THIS IS WHAT LOGS OUT OF MY FUNCTION
info: User function triggered, starting execution
info: Execution took 2034 ms, finished with status: 'crash'

And this below is what my client gets: the error message is simply internal, because cloud functions avoid passing internal error details to the client.

enter image description here

Note4: I haven't deployed the function yet. This is all on local development.

Upvotes: 3

Views: 3048

Answers (2)

cbdeveloper
cbdeveloper

Reputation: 31365

I have managed to do it.

According to this Firebase youtube video, the os.tmpdir() only one directory that is writable by your function's code.

So did the following and now it works:

index.js

const fs = require('fs');
const os = require('os');
const path = require('path');
const tmpdir = os.tmpdir();
const filePath = path.join(tmpdir,'response.html');

// PROMISIFY WRITEFILE TO SAVE URL RESPONSE
const promisifiedWriteFile = function(text) {
  return new Promise((resolve, reject) => {
    fs.writeFile(filePath, text, err => {
      if (err) {
        return reject(err);
      }
      return resolve();
    });
  });
};

And from inside the function I'm calling:

await promisifiedWriteFile(response.body);

Found some related doc also on this link:

https://cloud.google.com/functions/docs/concepts/exec

Upvotes: 9

user835611
user835611

Reputation: 2366

I believe you're not allowed to write to the filesystem on Firebase functions. Try writing to /tmp, see Does Cloud Functions for Firebase support file operation?.

But I think you shouldn't be writing response.html, instead you should set the text and status code in the response that the Firebase function returns.

Upvotes: 2

Related Questions