CODEWITHSUNDEEP
javascriptnode.js
dev
dev

Reputation: 11329

Node.js file write in loop fails randomly

Here is my code :

function aCallbackInLoop(dataArray) {
        dataArray.forEach(function (item, index) {

            fs.appendFile(fileName, JSON.stringify(item) + "\r\n", function (err) {
                if (err) {
                    console.log('Error writing data ' + err);
                } else {
                    console.log('Data written');
                }
            });
        });
    }

I get random errors :

Data written
Data written
.
.
Error writing data Error: UNKNOWN, open 'output/mydata.json'
Error writing data Error: UNKNOWN, open 'output/mydata.json'
.
.
Data written
Error writing data Error: UNKNOWN, open 'output/mydata.json'

The function (aCallbackInLoop) is a callback for a web-service request, which returns chunks of data in dataArray. Multiple web-service requests are being made in a loop, so this callback is perhaps being called in parallel. I doubt it's some file lock issue, but I am not sure how to resolve.

PS: I have made sure it's not a data issue (I am logging all items in dataArray)

Edit : Code after trying write stream :

function writeDataToFile(fileName, data) {
    try {
        var wStream = fs.createWriteStream(fileName);
        wStream.write(JSON.stringify(data) + "\r\n");
        wStream.end();
    } catch (err) {
        console.log(err.message);
    }
}

function aCallbackInLoop(dataArray){
    dataArray.forEach(function(item, index){
        writeDataToFile(filename, item);   //filename is global var
    });
}

Upvotes: 6

Views: 3563

Answers (2)

thefourtheye
thefourtheye

Reputation: 239683

As you have observed, multiple appendFile calls are not able to proceed because of the previous appendFile calls. In this particular case, it would be better to create a write stream.

var wstream = fs.createWriteStream(fileName);

dataArray.forEach(function (item) {
    wstream.write(JSON.stringify(item + "\r\n");
});

wstream.end();

If you want to know when all the data is written, then you can register a function with the finish event, like this

var wstream = fs.createWriteStream(fileName);

wstream.on("finish", function() {
    // Writing to the file is actually complete.
});

dataArray.forEach(function (item) {
    wstream.write(JSON.stringify(item + "\r\n");
});

wstream.end();

Upvotes: 8

manojlds
manojlds

Reputation: 301587

Try using the synchronous version of appendFile - https://nodejs.org/api/fs.html#fs_fs_appendfilesync_filename_data_options

Upvotes: 2

Related Questions