CODEWITHSUNDEEP
javascriptnode.jsfs
jabuj
jabuj

Reputation: 3639

Nodejs: synchronizing multiple simultaneous file changes

I have a small development web server, that I use to write missing translations into files.

app.post('/locales/add/:language/:namespace', async (req, res) => {
  const { language, namespace } = req.params
  // I'm using fs.promises
  let current = await fs.readFile(`./locales/${language}/${namespace}.json`, 'utf8')
  current = JSON.parse(current)

  const newData = JSON.stringify({ ...req.body, ...current }, null, 2)
  await fs.writeFile(`./locales/${language}/${namespace}.json`, newData)
})

Obviously, when my i18n library does multiple writes into one file like this:

fetch('/locales/add/en/index', { body: `{"hello":"hello"}` })
fetch('/locales/add/en/index', { body: `{"bye":"bye"}` })

it seems like the file is being overwritten and only the result of the last request is saved. I cannot just append to the file, because it's JSON. How to fix this?

Upvotes: 2

Views: 1297

Answers (1)

jfriend00
jfriend00

Reputation: 707158

You will have to use some sort of concurrency control to keep two concurrent requests that are both trying to write to the same resources form interfering with each other.

If you have lots of different files that you may be writing to and perhaps multiple servers writing to it, then you pretty much have to use some sort of file locking, either OS-supplied or manually with lock files and have subsequent requests wait for the file lock to be cleared. If you have only on server writing to the file and a manageable number of files, then you can create a file queue that keeps track of the order of requests and when the file is busy and it can return a promise when it's time for a particular request to do its writing

Concurrency control is always what databases are particularly good at.

I have no experience with either of these packages, but these are the general idea:

https://www.npmjs.com/package/lockfile

https://www.npmjs.com/package/proper-lockfile

These will guarantee one at a time access. I don't know if they will guarantee that multiple requests are granted access in the precise order they attempted to acquire the lock. If you need that, you might have to add that on top with some sort of queue.

Some discussion of this topic here: How can I lock a file while writing to it asynchronously

Upvotes: 2

Related Questions