CODEWITHSUNDEEP
node.jsfs
drmrbrewer
drmrbrewer

Reputation: 12989

Why is fs.readFileSync() faster than await fsPromises.readFile()?

Here is the test code (in an express environment just because that's what I happen to be messing around with):

const fs = require('fs-extra');
const fsPromises = fs.promises;
const express = require('express');
const app = express();

const speedtest = async function (req, res, next) {
    const useFsPromises = (req.params.promises == 'true');
    const jsonFileName = './json/big-file.json';
    const hrstart = process.hrtime();
    if (useFsPromises) {
        await fsPromises.readFile(jsonFileName);
    } else {
        fs.readFileSync(jsonFileName);
    }
    res.send(`time taken to read: ${process.hrtime(hrstart)[1]/1000000} ms`);
};

app.get('/speedtest/:promises', speedtest);

The big-file.json file is around 16 MB. Using node 12.18.4.

Typical results (varies quite a bit around these values, but the following are "typical"):

https://dev.mydomain.com/speedtest/false
time taken to read: 3.948152 ms

https://dev.mydomain.com/speedtest/true
time taken to read: 61.865763 ms

UPDATE to include two more variants... plain fs.readFile() and also a promisified version of this:

const fs = require('fs-extra');
const fsPromises = fs.promises;
const util = require('util');
const readFile = util.promisify(fs.readFile);
const express = require('express');
const app = express();

const speedtest = async function (req, res, next) {
    const type = req.params.type;
    const jsonFileName = './json/big-file.json';
    const hrstart = process.hrtime();
    if (type == 'readFileFsPromises') {
        await fsPromises.readFile(jsonFileName);
    } else if (type == 'readFileSync') {
        fs.readFileSync(jsonFileName);
    } else if (type == 'readFileAsync') {
        return fs.readFile(jsonFileName, function (err, jsondata) {
            res.send(`time taken to read: ${process.hrtime(hrstart)[1]/1000000} ms`);
        });
    } else if (type == 'readFilePromisified') {
        await readFile(jsonFileName);
    }
    res.send(`time taken to read: ${process.hrtime(hrstart)[1]/1000000} ms`);
};

app.get('/speedtest/:type', speedtest);

I am finding that the fsPromises.readFile() is the slowest, while the others are much faster and all roughly the same in terms of reading time. I should add that in a different example (which I can't fully verify so I'm not sure what was going on) the time difference was vastly bigger than reported here. Seems to me at present that fsPromises.readFile() should simply be avoided because there are other async/promise options.

Upvotes: 12

Views: 5170

Answers (1)

jfriend00
jfriend00

Reputation: 707318

After stepping through each implementation in the debugger (fs.readFileSync and fs.promises.readFile), I can confirm that the synchronous version reads the entire file in one large chunk (the size of the file). Whereas fs.promises.readFile() reads 16,384 bytes at a time in a loop, with an await on each read. This is going to make fs.promises.readFile() go back to the event loop multiple times before it can read the entire file. Besides giving other things a chance to run, it's extra overhead to go back to the event loop every cycle through a for loop. There's also memory management overhead because fs.promises.readFile() allocates a series of Buffer objects and then combines them all at the end, whereas fs.readFileSync() allocates one large Buffer object at the beginning and just reads the entire file into that one Buffer.

So, the synchronous version, which is allowed to hog the entire CPU, is just faster from a pure time to completion point of view (it's significantly less efficient from a CPU cycles used point of view in a multi-user server because it blocks the event loop from doing anything else during the read). The asynchronous version is reading in smaller chunks, probably to avoid blocking the event loop too much so other things can effectively interleave and run while fs.promises.readFile() is doing its thing.

For a project I worked on awhile ago, I wrote my own simple asynchronous version of readFile() that reads the entire file at once and it was significantly faster than the built-in implementation. I was not concerned about event loop blockage in that particular project so I did not investigate if that's an issue.

In addition, fs.readFile() reads the file in 524,288 byte chunks (much larger chunks that fs.promises.readFile()) and does not use await, using just plain callbacks. It is apparently just coded more optimally than the promise implementation. I don't know why they rewrote the function in a slower way for the fs.promises.readFile() implementation. For now, it appears that wrapping fs.readFile() with a promise would be faster.

Upvotes: 10

Related Questions