CODEWITHSUNDEEP
javascriptnode.jsrequestnpm
CRQ
CRQ

Reputation: 357

Error: EMFILE: too many open files

Using nw.js, I am just trying to save images in an array of img elements with different random names.

But having a few errors, is something wrong with my code?

for (i = 0; i < imgs.length; i++) {
 request(imgs[i].getAttribute('src')).on('error', function(err) {
  throw err
 }).pipe(fs.createWriteStream('data/imgs/' + randomString))
}

imgs[] is an array of 100-500 html img element, but I am receiving 

Error: EMFILE: too many open files, open *<directory>*

And another error:

"Uncaught Error: socket hang up"

Although it saves some images, some of them are corrupted, and it creates too many images than there actually is.

Upvotes: 6

Views: 11214

Answers (2)

thefourtheye
thefourtheye

Reputation: 239653

Use graceful-fs module, which is a drop-in replacement for the fs module. It is a wrapper around the native fs module. Quoting the docs of graceful-fs module,

Queues up open and readdir calls, and retries them once something closes if there is an EMFILE error from too many file descriptors.

Since it exposes the same APIs as the native fs module, you can use it just like the normal fs module.

// use just like fs
var fs = require('graceful-fs');

Note: This library was created by, Isaac Z. Schlueter, one of the core developers of the Node.js.

Upvotes: 10

Ingenium
Ingenium

Reputation: 29

Another simple solution would be just to wait for file to be downloaded and only then open new request by using recursion:

var fs = require('fs');
var request = require('request');

var dest = '../data/downloads/';

function _saveAllFiles (fileUrlArray, curIdx, options) {
    try {
        var count = fileUrlArray.length - 1;
        var curFile = fileUrlArray[curIdx];

        var stream = request(curFile)
            .pipe(fs.createWriteStream(dest + "file_" + curIdx));

        stream.on('finish', function () {
            console.log("Downloaded", curFile);
            stream.close();
            if (curIdx + 1 < count) {
                //Finished, download next file
                _saveAllFiles(fileArray, curIdx + 1, options);
            }
        });
        stream.on('error', function () {
            console.log("Error", curFile);
            stream.close();
            stream.end();
            if (curIdx + 1 < count) {
                //Error occured, go to next file
                _saveAllFiles(fileArray, curIdx + 1, options);
            }
        });
    } catch (err) {
        console.error("Failed to download file", err);
    }
}

Upvotes: 0

Related Questions