CODEWITHSUNDEEP
node.js
Sina Mirhejazi
Sina Mirhejazi

Reputation: 322

NodeJS - read and write file causes corruption

I'm kinda new to NodeJS and I'm working on a simple file encoder.

I planned to change the very first 20kb of a file and just copy the rest of it.

So I used the following code, but it changed some bytes in the rest of the file.

Here is my code:

var fs = require('fs');
var config = require('./config');


fs.open(config.encodeOutput, 'w', function(err, fw) {
     if(err) {
        console.log(err);
    } else {
        fs.readFile(config.source, function(err, data) {
            var start = 0;
            var buff = readChunk(data, start);
            while(buff.length) {
                if(start < config.encodeSize) {
                    var buffer = makeSomeChanges(buff);

                    writeChunk(fw, buffer);
                } else {
                    writeChunk(fw, buff);
                }

                start += config.ENCODE_BUFFER_SIZE;
                buff = readChunk(data, start);
            }
        });
    }  
});

function readChunk(buffer, start) {
    return buffer.slice(start, start + config.ENCODE_BUFFER_SIZE);
}

function writeChunk(fd, chunk) {
    fs.writeFile(fd, chunk, {encoding: 'binary', flag: 'a'});
}

I opened encoded file and compared it with the original file.

I even commented these parts:

//if(start < config.encodeSize) {
//    var buffer = makeSomeChanges(buff);

//    writeChunk(fw, buffer);
//} else {
    writeChunk(fw, buff);
//}

So my program just copies the file, but it still changes some bytes.

What is wrong?

Upvotes: 1

Views: 2488

Answers (2)

Davide Cannizzo
Davide Cannizzo

Reputation: 3134

Since you were using asynchronous IO, you should've been waiting for a queue of operations, as multiple writes happening at the same time are likely to end up corrupting your file. This explains why your issue is solved using synchronous IO — this way, a further write cannot start before the previous one completed.

However, using synchronous APIs when asynchronous ones are available is a poor choice, due to which your program will be actually blocked while it writes to the file. You should go for async and create a queue to wait for.

Upvotes: 3

Sina Mirhejazi
Sina Mirhejazi

Reputation: 322

So I checked the pattern and I realized some bytes are not in the right place and I guessed that it should be because I'm using async write function.

I changed fs.writeFile() to fs.writeFileSync() and everything is working fine now.

Upvotes: 3

Related Questions