CODEWITHSUNDEEP
node.js
Michael
Michael

Reputation: 9402

atomically appending to a log file in nodejs

NodeJS is asynchronous, so for example if running an Express server, one might be in the middle of servicing one request, log it, and then start servicing another request and try to log it before the first has finished.

Since these are log files, it's not a simple write. Even if a write was atomic, maybe another process actually winds up writing at the offset the original process is about to and it winds up overwriting.

There is a synchronous append function (fs.appendFile) but this would require us to delay servicing a request to wait for a log file write to complete and I'm still not sure if that guarantees an atomic append. What is the best practice for writing to log files in NodeJS while ensuring atomicitiy?

Upvotes: 2

Views: 724

Answers (1)

Peter Lyons
Peter Lyons

Reputation: 146014

one might be in the middle of servicing one request, log it, and then start servicing another request and try to log it before the first has finished.

The individual write calls will be atomic, so as long as you make a single log write call per request, you won't have any corruption of log messages. It is normal, however if you log multiple messages while processing a request, for those to be interleaved between many different concurrent requests. Each message is intact, but they are in the log file in chronological order, not grouped by request. That is fine. You can filter on a request UUID if you want to follow a single request in isolation.

Even if a write was atomic, maybe another process actually winds up writing at the offset the original process is about to and it winds up overwriting.

Don't allow multiple processes to write to the same file or log. Use process.stdout and all will be fine. Or if you really want to log directly to the filesystem, use an exclusive lock mechanism.

What is the best practice for writing to log files in NodeJS while ensuring atomicitiy?

process.stdout, one write call per coherent log message. You can let your process supervisor (systemd or upstart) write your logs for you, or use a log manager such as multilog, sysvlogd and pipe your stdout to them and let them handle writing to disk.

Upvotes: 2

Related Questions