CODEWITHSUNDEEP
javascriptnode.jspromiseasync-awaitfs
apena
apena

Reputation: 2331

What exactly is affecting how fs.writeFile behaves when using async/await in a loop requiring a callback such as Array.map?

This issue appears to be related to string length but I have seen it occur when including a console.log() in certain places of the code. To be clear, this issue only occurs when using fs.writeFile in a loop that requires a callback such as Array.map.

I am able to loop through an array and make a series of (asynchronous) fs.WriteFile calls with strings of seemingly any size with as many console.log() calls as I like when I use a for loop and there is no issue. The expected result is that, by the end of the loop, the file written to contains only the data from the last element of the array. Everything works great. You can check it out on here on repl.it. And here is the code:

const fs = require('fs')

const writeFile = (uri, data, options) => new Promise((resolve, reject) => {
  fs.writeFile(uri, data, (err) => {
    if (err) {
      return reject(`Error writing file: ${uri} --> ${err}`)
    }
    resolve(`Successfully wrote file: ${uri}  --> with data: ${data}`)
  })
})

const asyncWriteFile = (uri, data) => {
  return writeFile(uri, data).then( res => console.log(res)).catch(e => console.log(e))
}
const asyncWriteFileLoop = async (uri, seed) => {
  for (let i = 0; i < dataArr.length; i++) {
    await asyncWriteFile(uri, seed[i])
  }
}

const uri = 'test-write-dump.txt'
const dataArr = [
  'this is string long enough to cause an issue when using Array.map but not in a for loop',[1,2,3,4],1600,'foobar',['foo','bar','baz']
]

// works as expected, output in file is foo,bar,baz
asyncWriteFileLoop(uri, dataArr)

Now here is the issue: As soon as I try to replace the for loop with Array.map using Promise.all. The file is appended rather than overwritten and in a seemingly odd and unpredictable order. I realize this is the nature of calling fs.writeFile too quickly in a row. However I was expecting the array of promises to resolve one after the other thus circumventing the issue but this appears not to be the case. You can check it out on here on repl.it. Here is the offending code:

const fs = require('fs')

const writeFile = (uri, data, options) => new Promise((resolve, reject) => {
  fs.writeFile(uri, data, (err) => {
    if (err) {
      return reject(`Error writing file: ${uri} --> ${err}`)
    }
    resolve(`Successfully wrote file: ${uri}  --> with data: ${data}`)
  })
})

const appendFile = (uri, data, options) => new Promise((resolve, reject) => {
  fs.appendFile(uri, data, (err) => {
    if (err) {
      return reject(`Error appending file: ${uri} --> ${err}`)
    }
    resolve(`Successfully appended file: ${uri}  --> with data: ${data}`)
  })
})

asyncWriteFileMap = async (uri, seed) => {
  const promises = seed.map(async data => {
    await writeFile(uri, data)
      .then(res => console.log(res))
      .catch(e => console.log(e))
  })
  const result = await Promise.all(promises)
}

const uri = 'test-write-dump.txt'
const dataArr = [
  'This is a string long enough to cause an issue, change this to a single character and the expected result almost always occurs',[1,2,3,4],1600,'foobar',['foo','bar','baz']
]

// Seems like it doesnt work with long strings, writes a junk file
asyncWriteFileMap(uri, dataArr)

I would really like to know exactly what is causing this behavour, why for loops work and Array.map doesn't.

Upvotes: 0

Views: 361

Answers (1)

jfriend00
jfriend00

Reputation: 708036

.map() is NOT promise-aware. It does not pay any attention to the promise that your callback returns. Instead, it just puts the promise in the resulting array and continues with its loop.

So, what happens is that .map() starts all the writeFileAsync() calls at once and then sometime later each of those promises finishes. The await you're using inside the callback does not pause the .map() iteration - in fact, it doesn't really help you at all.

The for loop, on the other hand, is promise-aware and will pause the loop for the await.

So, the for loop is paused by the await, the .map() loop is not.

In case, you didn't realize, when an async function (like your .map() callback) encounters its first await, the async function immediately returns a promise. It suspends further function execution until the await sees a resolved promise, but the function itself returns a promise and execution of the calling code continues. That's why .map() continues with the rest of its iterations and does not wait for the await.

Also, keep in mind that Promise.all() doesn't "run" anything. The asynchronous operations are already running and Promise.all() is just given an array of promises to monitor and collect results from. So, it's not correct to say that Promise.all() runs in parallel, not in a specific order. It is .map() that is doing that in your example.

Promise.all() is usually used in circumstances when multiple promise-based asynchronous operations are being run in parallel, but those operations are started by something else. Promise.all() is just monitoring the completion and collecting results.

Upvotes: 3

Related Questions