CODEWITHSUNDEEP
c#restamazon-s3
sam360
sam360

Reputation: 1131

Download multiple objects with a single operation from Amazon S3

Using Amazon REST API, is there way to download multiple files perhaps in one Bucket with a single operation? For example to zip all the objects and download them as one file?

Note that I am looking for operation efficiency and than custom solutions. For example of I was to download each individual file from S3 and Create the ZIP file myself the operation could take up to 40 seconds for 100 files of 1.5MB size , I am looking for a solution that allow me to do this in less 10 seconds.

Upvotes: 0

Views: 1745

Answers (1)

Michael - sqlbot
Michael - sqlbot

Reputation: 179064

There is no REST call that can do this in S3.

As you no doubt realize, the transfer bandwidth doesn't likely account for much of the wall-clock time your process is taking.

The most likely solution would be to use either multiple threads or asynchronous I/O (depending on your language and environment) to send the requests to S3 in parallel groups, combining the results when all of the desired objects have been successfully fetched.

Of course, if what you are doing, now, does not reuse the user agent's connection so that you can take advantage of HTTP keep-alives, you should see some level of performance improvement if you can enable that functionality, by avoiding unnecessarily repeated connection setups (and potentially, SSL negotiations).

Upvotes: 1

Related Questions