Is there a more elegant way to read then write *the same file* with node js stream

Question

I wanna read file then change it with through2 then write into the same file, code like:

const rm = require('rimraf')
const through2 = require('through2')
const fs = require('graceful-fs')
// source file path
const replacementPath = `./static/projects/${destPath}/index.html`
// temp file path
const tempfilePath = `./static/projects/${destPath}/tempfile.html`
// read source file then write into temp file
await promiseReplace(replacementPath, tempfilePath)    
// del the source file
rm.sync(replacementPath)
// rename the temp file name to source file name
fs.renameSync(tempfilePath, replacementPath)
// del the temp file
rm.sync(tempfilePath)

// promiseify readStream and writeStream
function promiseReplace (readfile, writefile) {
  return new Promise((res, rej) => {
    fs.createReadStream(readfile)
      .pipe(through2.obj(function (chunk, encoding, done) {
        const replaced = chunk.toString().replace(/id="wrap"/g, 'dududud')
        done(null, replaced)
      }))
      .pipe(fs.createWriteStream(writefile))
      .on('finish', () => {
        console.log('replace done')
        res()
      })
      .on('error', (err) => {
        console.log(err)
        rej(err)
      })
  })
}

the above code works, but I wanna know can I make it more elegant ?

and I also try some temp lib like node-temp

unfortunately， it cannot readStream and writeStream into the same file as well, and I open a issues about this.

So any one know a better way to do this tell me, thank you very much.

Answer 1

Streams are useless in this situation, because they return you chunks of file that can break the string that you're searching for. You could use streams, then merge all these chunks to get content, then replace the string that you need, but that will be longer code that will provoke just one question: why do you read file by chunks if you don't use them ?

The shortest way to achieve what you want is:

let fileContent = fs.readFileSync('file_name.html', 'utf8')
let replaced = fileContent.replace(/id="wrap"/g, 'dududud')
fs.writeFileSync('file_name.html', replaced)

All these functions are synchronous, so you don't have to promisify them

Answer 2

You can make the code more elegant by getting rid of unnecessary dependencies and using the newer simplified constructor for streams.

const fs = require('fs');
const util = require('util');
const stream = require('stream');
const tempWrite = require('temp-write');

const rename = util.promisify(fs.rename);

const goat2llama = async (filePath) => {
    const str = fs.createReadStream(filePath, 'utf8')
        .pipe(new stream.Transform({
            decodeStrings : false,
            transform(chunk, encoding, done) {
                done(null, chunk.replace(/goat/g, 'llama'));
            }
        }));
    const tempPath = await tempWrite(str);
    await rename(tempPath, filePath);
};

Tests

AVA tests to prove that it works:

import fs from 'fs';
import path from 'path';
import util from 'util';
import test from 'ava';
import mkdirtemp from 'mkdirtemp';
import goat2llama from '.';

const writeFile = util.promisify(fs.writeFile);
const readFile = util.promisify(fs.readFile);

const fixture = async (content) => {
    const dir = await mkdirtemp();
    const fixturePath = path.join(dir, 'fixture.txt');
    await writeFile(fixturePath, content);
    return fixturePath;
};

test('goat2llama()', async (t) => {
    const filePath = await fixture('I like goats and frogs, but goats the best');
    await goat2llama(filePath);
    t.is(await readFile(filePath, 'utf8'), 'I like llamas and frogs, but llamas the best');
});

A few things about the changes:

• Through2 is not really needed anymore. It used to be a pain to set up passthrough or transform streams properly, but that is not the case anymore thanks to the simplified construction API.
• You probably don't need graceful-fs, either. Unless you are doing a lot of concurrent disk I/O, EMFILE is not usually a problem, especially these days as Node has gotten smarter about file descriptors. But that library does help with temporary errors caused by antivirus software on Windows, if that is a problem for you.
• You definitely do not need rimraf for this. You only need fs.rename(). It is similar to mv on the command line, with a few nuances that make it distinct, but the differences are not super important here. The point is there will be nothing at the temporary path after you rename the file that was there.
• I used temp-write because it generates a secure random filepath for you and puts it in the OS temp directory (which automatically gets cleaned up now and then), plus it handles converting the stream to a Promise for you and takes care of some edge cases around errors. Disclosure: I wrote the streams implementation in temp-write. :)

Overall, this is a decent improvement. However, there remains the boundary problem discussed in the comments. Luckily, you are not the first person to encounter this problem! I wouldn't call the actual solution particularly elegant, certainly not if you implement it yourself. But replacestream is here to help you.

const fs = require('fs');
const util = require('util');
const tempWrite = require('temp-write');
const replaceStream = require('replacestream');

const rename = util.promisify(fs.rename);

const goat2llama = async (filePath) => {
    const str = fs.createReadStream(filePath, 'utf8')
        .pipe(replaceStream('goat', 'llama'));
    const tempPath = await tempWrite(str);
    await rename(tempPath, filePath);
};

Also...

I do not like temp files

Indeed, temp files are often bad. However, in this case, the temp file is managed by a well-designed library and stored in a secure, out-of-the-way location. There is virtually no chance of conflicting with other processes. And even if the rename() fails somehow, the file will be cleaned up by the OS.

That said, you can avoid temp files altogether by using fs.readFile() and fs.writeFile() instead of streaming. The former also makes text replacement much easier since you do not have to worry about chunk boundaries. You have to choose one approach or the other, however for very big files, streaming may be the only option, aside from manually chunking the file.