CODEWITHSUNDEEP
javascriptnode.jsrecursion
iros
iros

Reputation: 137

How to prevent javascript from opening too many files at once?

I'm currently learning javascript and nodeJS by writing a simple music database program. The idea is pretty simple, it recursively scans a directory, parses any mp3 files, and makes a database from this info. I wrote my first version and it looked like this:

var fs = require('graceful-fs'); 
var metadata = require('musicmetadata'); 


function recursiveScan(directory){
    var files = fs.readdirSync( directory );

    // loop through files
    for (var i=0; i < files.length; i++) {
        var filePath = directory + files[i];
        var stat = fs.statSync(filePath);

        // Check if it's a directory
        if(stat.isDirectory()){
            // Scan that directory
            recursiveScan(filePath + '/');
        }else{
            // Find the file extension
            var extension = getFileExtension(files[i]);

            // If it's an mp3 file, parse it and add it the db
            if (extension == 'mp3' ) {
                var parser = metadata(fs.createReadStream(filePath), function (err, songInfo) {
                  // ... insert song into database

                });
            }
        }
    }
}

I ran it on a directory with a few dozen mp3s and it worked fine. Then I ran it on a directory with a few thousand mp3s and quickly realized the fatal flaw.

Due to the metadata plugins's asynchronous nature, this script will recursively scan all the directories really quickly and try to open up too many files at once causing the it to crash.

What would be the best way to limit the number of files open at a given time? I was thinking about using a generator, but I'm still trying to wrap my head around how they work.

Upvotes: 2

Views: 142

Answers (1)

Lakmal Vithanage
Lakmal Vithanage

Reputation: 2777

Use promises to do that. It ensure the flow of the code and prevent loading many files into memory once. A good explination can be find here

Upvotes: 1

Related Questions