CODEWITHSUNDEEP
javascriptnode.js
Oscar J. Irun
Oscar J. Irun

Reputation: 475

Copying large amount of files with nodejs?

I have to copy a large amount of files (like 25000) asynchronously. I'm using this library: https://github.com/stephenmathieson/node-cp.

This is my code:

         for(var i = 0; i < 25000; i++){
            cp(origin[i], dest[i], function(err){
              console.log("successfully copied")
            })
         }

It completes the loop but it doens't copy every item. The "Successfully copied" is called between 6000 and 8000 times. After that it doesn't copy anymore. It has something to do with the memory or a limit for async tasks?

Any help would be appreciated!

Upvotes: 1

Views: 465

Answers (2)

Alongkorn
Alongkorn

Reputation: 4207

you can copy async with this,

var fs = require('fs-extra')

fs.copy('/tmp/myfile', '/tmp/mynewfile', function (err) {
  if (err) return console.error(err)
  console.log("success!")
}) // copies file 

fs.copy('/tmp/mydir', '/tmp/mynewdir', function (err) {
  if (err) return console.error(err)
  console.log('success!')
}) // copies directory, even if it has subdirectories or file

info => https://www.npmjs.com/package/fs-extra

Upvotes: 0

Jack Guy
Jack Guy

Reputation: 8523

The copy function takes a callback which is usually a good clue that it's asynchronous. That means that the for loop will continue to run even though the copy hasn't completed, meaning you just queued up 25,000 copy operations!

There's a few ways to solve this but among the most common is using the async module.

var async = require('async');
async.forEachOf(origin, function (file, i, callback) {
    cp(file, dest[i], function(err){
        callback();
    });
})

This won't proceed to the next iteration of the loop until callback is called.

Upvotes: 2

Related Questions