CODEWITHSUNDEEP
node.jsstreamgunzip
metalaureate
metalaureate

Reputation: 7732

node.js how to stream 22GB gzipped file uncompress on the fly

I need to iterate over the lines in a 22GB gzipped file. Expanded, the file is 250GB.(It's the Google Freebase db dump)

I'm not too savvy with streams. Is there a way to stream the file and gunzip it as I go?

Here is going the other way:

var r = fs.createReadStream('file.txt');
var z = zlib.createGzip();
var w = fs.createWriteStream('file.txt.gz');
r.pipe(z).pipe(w);

Upvotes: 0

Views: 3784

Answers (1)

Dan D.
Dan D.

Reputation: 74685

Here is the reverse operation using zlib.createGunzip instead of zlib.createGzip:

var r = fs.createReadStream('file.txt.gz');
var z = zlib.createGunzip();
var w = fs.createWriteStream('file.txt');
r.pipe(z).pipe(w);

Upvotes: 9

Related Questions