CODEWITHSUNDEEP
node.jsamazon-web-servicesamazon-s3
Meto
Meto

Reputation: 440

AWS S3 Node.js SDK: Need to Download all files from one bucket at once

Currently I can iterate over all files in bucket and download one by one, using node.js sdk. I need to download all files from S3 bucket at once. then remove it from bucket.

Upvotes: 3

Views: 14732

Answers (3)

zya
zya

Reputation: 850

You can use the aws cli using the command below: 

aws s3 sync s3://yourbucket .

If you have to use node then you can use exec or spwan.

const exec = require('child_process').exec;
exec('aws s3 sync s3://yourbucket .', (err, stdout, stderr) => {});

Upvotes: 2

Kyle Mathews
Kyle Mathews

Reputation: 3278

This higher-level Node S3 library has a downloadDir function which syncs a remote S3 bucket with a local directory.

https://github.com/andrewrk/node-s3-client#clientdownloaddirparams

Upvotes: 4

Max
Max

Reputation: 8836

Let's say you have a function listFiles that returns an array of the Keys in the bucket you want to download and a function downloadFile that accepts a file Key (name) and downloads it and then deletes it.

You want to call downloadFile simultaneously on all files returned from listFiles right? Use the async library.

function listFiles(cb) {
    s3.stuff(params, cb);
}
function downlaodFile(key, cb) {
    s3.stuff(key, cb);
}
listFiles(function (err, fileKeys) {
    if (err) {
        throw err;//don't really but this is just an example
    }
    async.each(fileKeys, downloadFile, function done(err) {
        if (err) {
            throw err;
        }
    });
});

Upvotes: 0

Related Questions