CODEWITHSUNDEEP
node.jsfilefs
Dom Vinyard
Dom Vinyard

Reputation: 2088

Node .fs Working with a HUGE Directory

Picture a directory with a ton of files. As a rough gauge of magnitude I think the most that we've seen so far is a couple of million but it could technically go another order higher. Using node, I would like to read files from this directory, process them (upload them, basically), and then move them out of the directory. Pretty simple. New files are constantly being added while the application is running, and my job (like a man on a sinking ship holding a bucket) is to empty this directory as fast as it's being filled.

So what are my options? fs.readdir is not ideal, it loads all of the filenames into memory which becomes a problem at this kind of scale. Especially as new files are being added all the time and so it would require repeated calls. (As an aside for anybody referring to this in the future, there is something being proposed to address this whole issue which may or may not have been realised within your timeline.)

I've looked at the myriad of fs drop-ins (graceful-fs, chokadir, readdirp, etc), none of which have this particular use-case within their remit.

I've also come across a couple of people suggesting that this can be handled with child_process, and there's a wrapper called inotifywait which tasks itself with exactly what I am asking but I really don't understand how this addresses the underlying problem, especially at this scale.

I'm wondering if what I really need to do is find a way to just get the first file (or, realistically, batch of files) from the directory without having the overhead of reading the entire directory structure into memory. Some sort of stream that could be terminated after a certain number of files had been read? I know Go has a parameter for reading the first n files from a directory but I can't find a node equivalent, has anybody here come across one or have any interesting ideas? Left-field solutions more than welcome at this point!

Upvotes: 2

Views: 494

Answers (1)

Aminadav Glickshtein
Aminadav Glickshtein

Reputation: 24590

You can use your operation system listing file command, and stream the result into NodeJS.

For example in Linux:

var cp=require('child_process')
var stdout=cp.exec('ls').stdout

stdout.on('data',function(a){
         console.log(a)
});0

RunKit: https://runkit.com/aminanadav/57da243180f3bb140059a31d

Upvotes: 1

Related Questions