Is it possible to write to a file from two threads at once in Node.js?

Question

I found this question more or less asked elsewhere, in less-than-clear language, but the answer was

"I doubt you have any reason to."

which doesn't really seem like a valid answer, considering if it's fundamentally possible (which it very well may not be), it would allow us to generate large (100GB+) files in parallel threads much faster. Now, if your data structure is something that could really be written in such a parallel way, you can probably just write 16 different files on your 16 threads and call it a day. That's probably what I'll be doing.

But just out of curiosity, let's say we really, really want all that data stored in one file. Can multiple

fs.createWriteStream(filePath)

be called from separate threads and successfully write to locations within the same file in parallel?

I couldn't find any info on this in the documentation, although I suspect the answer is simply "no."

Answer 1

If you really need this, I would suggest using a SQLite database, it's much safer (and faster) =)

Answer 2

You cannot safely append to the same file from two separate threads at the same time. Each thread will have its own file handle and its own write position and there are concurrency issues with where each file is writing the data to.

Can multiple fs.createWriteStream(filePath)be called from separate threads and successfully write to locations within the same file in parallel?

No. You cannot do that safely. Each thread will have concurrency issues keeping track of where the write position should be set to. You would need some concurrency management to sequence the writes and keep track of where each one was being written to.

You could have just one thread doing all the writing and have each thread that is generating data send its data to the one thread that is doing the writing as a means of safely sequencing the writes.