CODEWITHSUNDEEP
node.jsstream
Mas Bagol
Mas Bagol

Reputation: 4617

How to split a file to several parts, write those parts to disk and join them back with node js stream?

My goal is to create a node script for downloading using multiple connections. So, I plan to create a multiple request with range header, pipe the response stream to several parts, and then join all of them into single file when every connections is done downloading.

I'm not sure how to join those file parts. To make sure, i try write a half of file and write it with this:

const fs      = require('fs');

const input   = 'video.mp4';
const output1 = 'video.mp4.part1';

const half    = fs.statSync(input).size / 2;

const rstream = fs.createReadStream(input);
const part1 = fs.createWriteStream(output1);

part1
  .write(rstream.read(half));

But, there's an error happen:

TypeError: May not write null values to stream

I guess it's because the file not ended properly. So, how to do this (split and join a file back) correctly?

Upvotes: 1

Views: 4532

Answers (2)

Giorgio Bellisario
Giorgio Bellisario

Reputation: 191

You can split and merge file parts easily with node-split-file.

Here is a full working example of merging:

const fs = require('fs');
const splitFile = require('split-file');
const names = ["video.mp4.part1", "video.mp4.part2"];

splitFile.mergeFiles(names, __dirname + 'video.mp4')
  .then(() => {
    console.log('Done!');
  })
  .catch((err) => {
    console.log('Error: ', err);
  });

Upvotes: 0

Mohammad Abbas
Mohammad Abbas

Reputation: 574

You can specify the boundaries of the readable stream you create by providing the options 

{start , end}
. This will allow you to stream just the part you want to read from your original file.

You can simply add a listener to the data event and store the chunks into your writable stream.

here's the concept and you can build on it.

const fs      = require('fs');
const path    = require ('path');
const input   = 'video.mp4.zip';
const output1 = 'video.mp4.zip.part1';
const half    = fs.lstatSync(path.join(__dirname ,input)).size / 2;

const rstream = fs.createReadStream(path.join(__dirname ,input),
    {
        start : 0,
        end : half,
        autoClose: true
    }
);
writable = fs.createWriteStream(output1);
rstream.on ('data', (chunk) => {
    writable.write (chunk);
});

When reading the second part of the file, create your readable stream starting from the "half" position

const rstream2 = fs.createReadStream(path.join(__dirname ,input), 
    {
        start : half,
        autoClose: true 
    }
);

Also remember when combining the files back together, to specify the start position of your writable stream as well.

edit to add the combine code

As per the comment request, here is the combine method

const output   = 'combined.zip';
const input1 = 'video.mp4.zip.part1';
const input2 = 'video.mp4.zip.part2';
const file1Size =  fs.lstatSync(path.join(__dirname ,input1)).size;
const file2Size = fs.lstatSync(path.join(__dirname ,input2)).size;

const writable = fs.createWriteStream(output, {
  emitClose : true,
  autoClose: true
});
fs.createReadStream(path.join(__dirname ,input1), {
  autoClose : true
}).pipe(writable);

const writable2 = fs.createWriteStream(output, {
  start : file1Size,
  end : file2Size,
  autoClose: true
});
fs.createReadStream(input2, {
  autoClose: true
}).pipe(writable2);

Hope that helps

Upvotes: 4

Related Questions