CODEWITHSUNDEEP
node.js
miklosme
miklosme

Reputation: 791

Request makes process out of memory, when downloading big media files

I wrote a simple script to download video files from a CDN, where the direct URLs are simple to generate, for example http://something.com/N.mp4, where N is a number.

The problem is, when downloading files with larger than ~300MB, the files appears perfectly in hard drive, but before the request(...)'s callback, a memory allocation fail happens:

FATAL ERROR: CALL_AND_RETRY_0 Allocation failed - process out of memory

Does this happens because of some serious bad practice? Can request download media files, with this size?

Environment: Win7, 4GB+ free RAM, Node v0.10.31

var request = require('request');
var async = require('async');
var fs = require('fs');
var start = +process.argv[2] || 1;
var end = +process.argv[3] || 50;
var url = 'http://something.com/';

try {
  fs.mkdirSync(__dirname + '/videos/');
} catch (e) {}

var index = start;

async.whilst(
  function () { return index <= end; },
  function (callback) {
    var fileName = index + '.mp4';
    console.log('Started: ' + fileName);
    console.time('Done (' + fileName + ')');
    request(url + fileName, function() {
      console.timeEnd('Done (' + fileName + ')');
      index++;
      callback(null);
    }).pipe(fs.createWriteStream(__dirname + '/videos/' + fileName));
  },
  function (err) {
    if (err) {
      return console.error(err);
    }
    console.log('Script finished.');
  }
);

Example console output:

> node index.js 3
Started: 3.mp4
Done (3.mp4): 296592ms
Started: 4.mp4
Done (4.mp4): 369718ms
Started: 5.mp4
FATAL ERROR: CALL_AND_RETRY_0 Allocation failed - process out of memory

Upvotes: 4

Views: 1089

Answers (2)

vkurchatkin
vkurchatkin

Reputation: 13570

If you use request module with a callback it buffers the whole response body in memory. Try omitting callback and using finish event of fs stream instead.

var writer = fs.createWriteStream(__dirname + '/videos/' + fileName);
writer.on('finish', function() {
  // ...
  index++;
  callback(null);
});
request(url + fileName).pipe(writer);

Upvotes: 5

Andrew Lavers
Andrew Lavers

Reputation: 8151

It looks like you're trying to download videos 3 to 50 all in parallel, so that might be what's causing you to run out of memory. You could try doing them in series and see if that fixes the problem. With async.waterfall your code might look something like this:

var tasks = [];

for (; index < end; index++) {
    tasks.push(function(callback) {
        var fileName = index + '.mp4';
        console.log('Started: ' + fileName);
        console.time('Done (' + fileName + ')');
        request(url + fileName, function() {
            console.timeEnd('Done (' + fileName + ')');
            callback(null);
        }).pipe(fs.createWriteStream(__dirname + '/videos/' + fileName));
    });
}

async.waterfall(tasks, function(err) {
    if (err) {
        return console.error(err);
    }
    console.log('Script finished.');
});

Upvotes: -2

Related Questions