CODEWITHSUNDEEP
javascriptnode.js
ar2015
ar2015

Reputation: 6140

Node.js wait for all files read in the loop

I am new to the javascript/node.js event driven paradigm.

I need to stop the for after forEach to make sure all files have been read and then I continue. How should I implement wait_for_all_files_read() in this context?

my_list.forEach(function(element) {
  fs.readFile(element.file_path, function(err, data)  {
    if(err) throw err;
    element.obj=JSON.parse(data);
  });
});
wait_for_all_files_read(); <-----------
analyze(my_list)

Neither solution [1] or [2] work for me.

Upvotes: 2

Views: 3485

Answers (2)

RidgeA
RidgeA

Reputation: 1546

I can suggest you to rewrite the code to promise - it will make much easy to deal with it

const {promisisfy} = require('util')
const fs = require('fs')
const readFile = promisify(fs.readFile)

const fileNames = getFilenamesSomehow() // <-- returns array with path, e.g. ["./package.json", "/etc/hosts", "/etc/passwd"]

Promise.all(fileNames.map(name => readFile(name)))
.then(arrayWithFilesContent => analyze(arrayWithFilesContent))
.catch(err => handleError(err))

Next step what you may do - move the code to async/await functions

UPD

Assume you need read only one file, then parse it data to json and analyze the result somehow.

It this case you can do next:

readFile(singleFileName)
.then(function (singleFileContent) {
   return JSON.parse(singleFileContent)
})
.then(function (singleFileContentInJson) {
   return analyze(singleFileContentInJson)
})
.catch(funciton (error) {
  //here you can process all errors from functions above: reading file error, JSON.parse error, analyze error...
  console.log(error)
})

Then assume you need to analyze bunch of files

const fileNames = [...] // array with file names
// this create an array of promises, each of them read one file and returns the file content in JSON
const promises = fileNames.map(function (singleFileName) {
    return readFile(singleFileName)
    .then(function (singleFileContent) {
       return JSON.parse(singleFileContent)
    })
})

// promise all resolves (calls callback in 'then') all of promises in array are resolved and pass to then callback array with result of each promise
Promise.all(promises)
  .then(function (arrayWithResults) {
    return analyze(arrayWithResults)
  })
  // catch callback calls if one of promises in array rejects with error from the promise - so you can handle e.g. read file error or json parsing error here
  .catch(function (error) {
    //here you can handle any error
    console.log(error)
  })

Try to google some article to read how does promises work. E. g. you can start form mdn article

Upvotes: 4

Alex
Alex

Reputation: 4811

How I would do that:

  1. Promisify fs.readFile (by using, for example Bluebird)
  2. Mark the function as async
  3. Make a list of callbacks (my_list.map instead of forEach)
  4. "await Promise.all(myListOfCallbacks)"
  5. Next line after await will be executed after all the operations been finished

Something like that:

const {promisisfy} = require('util')
const fs = require('fs')
const readFile = promisify(fs.readFile)

const fileNames = getFilenamesArray();

async function executeMe() {
  try {
    const arrayWithFilesContent = await Promise.all(
      fileNames.map(name => readFile(name))
    );
    return analyze(arrayWithFilesContent);
  }
  catch (err) {
    handleError(err)
  }
}

executeMe();

Upvotes: 1

Related Questions