CODEWITHSUNDEEP
node.jshttpstream
MFave
MFave

Reputation: 1074

Node.js Streams: Is there a way to convert or wrap a fs write stream to a Transform stream?

With a node http server I'm trying to pipe the request read stream to the response write stream with some intermediary transforms, one of which is a file system write.

The pipeline looks like this with non pertinent code removed for simplicity: 

function handler (req, res) {
  req.pipe(jsonParse())
     .pipe(addTimeStamp())
     .pipe(jsonStringify())
     .pipe(saveToFs('saved.json'))
     .pipe(res);
}

The custom Transform streams are pretty straight forward, but I have no elegant way of writing saveToFs. It looks like this:

function saveToFs (filename) {
  const write$ = fs.createWriteStream(filename);
  write$.on('open', () => console.log('opened'));
  write$.on('close', () => console.log('closed'));

  const T = new Transform();
  T._transform = function (chunk, encoding, cb) {
    write$.write(chunk);
    cb(null, chunk);
  }
  return T;
}

The idea is simply to pipe the data to the write stream and then through to the response stream, but fs.createWriteStream(<file.name>) is only a writable stream, so it makes this approach difficult.

Right now this code has two problems that I can see: the write stream never fires a close event (memory leak?), and I would like the data to pass through the file system write before returning data to the response stream instead of essentially multicasting to two sinks.

Any suggestions, or pointing out fundamental things I've missed would be greatly appreciated.

Upvotes: 2

Views: 2868

Answers (1)

Marcos Casagrande
Marcos Casagrande

Reputation: 40374

What you should do is save the stream returned by the .pipe before saveToFs, and then pipe that to a file and res.

function handler(req, res) {
    const transformed = req.pipe(jsonParse())
        .pipe(addTimeStamp())
        .pipe(jsonStringify());

    transformed.pipe(fs.createWriteStream('saved.json'));
    transformed.pipe(res);
}

To sum it up, you can pipe the same readable stream (transformed) to multiple writable streams.

And I would like the data to pass through the file system write before returning data to the response stream instead of essentially multicasting to two sinks.

Use { end: false } option when piping to res.

transformed.pipe(res, { end: false });

And then call res.end() when the file is written or whenever you want.

Upvotes: 1

Related Questions