CODEWITHSUNDEEP
node.jsfs
Slava Knyazev
Slava Knyazev

Reputation: 6081

NodeJS read huge directory file by file

I have a directory composed of empty files and I want to iterate through each of their names. However I have nearly 20 million of these empty files and to load them all into memory with fs.readdir or fs.readdirSync would both take needlessly long and devour all my memory on the system.

What would be a way to go about this?

Ideally I would look for something that reads file by file in an async fashion with code that would ressemble the following:

readdirfilebyfile((filename)=>{....}) so that at no point would I keep the entire list of files in memory.

The current solution I am using is dumping all the file names into a single file which I then read as a data stream. However, this is just running away from a problem that I should know how to solve without resorting to this.

Upvotes: 2

Views: 1449

Answers (1)

dsdenes
dsdenes

Reputation: 1035

What about this one? pv is pipe viewer, a rate limiter for bash pipes.

const spawn = require('child_process').spawn;
const exec = require('child_process').exec;
const tail = spawn('tail -f /tpm/filelist | pv -l -L 10 -q');
tail.stdout.on('data', fileName => {
  // parse filenames here
  console.log(fileName);
});
exec('ls > /tpm/filelist');

Upvotes: 1

Related Questions