CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-lambda
Dev1ce
Dev1ce

Reputation: 5954

Is it possible to bulk download from AWS S3?

I wanted to know is it possible and if so how to bulk download from AWS S3?
From the AWS S3 documentation, it seems the getObject SDK method supports, getting only one object at a time

But there is a feature in AWS S3 called S3 batch operations

enter image description here

Not sure if the S3 batch operations support the use case of creating a job for downloading multiple objects as well?

The application I'm working on is using AWS Lambda,
And the bulk files I want to fetch are mostly images,
so the download operation is to be done on 100s of files, whose final zip size can go beyond 250 MB

Has anyone had such a use case?
If so how to achieve it?

Upvotes: 4

Views: 11357

Answers (2)

pfeilbr
pfeilbr

Reputation: 1949

This article has a potential good start by leveraging nodejs streams to ensure you don't hit the lambda memory and filesystem limits. Maybe wrap in step fn with checkpointing if time to complete becomes an issue.

Zip files on S3 with AWS Lambda and Node

Upvotes: 2

John Rotenstein
John Rotenstein

Reputation: 270264

No, this is not possible.

There is no API call to download multiple files. Instead, you will need to issue multiple API calls. You can multithread or use async in your application to download them faster.

Be careful: If your goal is to convert multiple files into a single Zip file, then you might exceed the disk storage space provided to the AWS Lambda function. There is only 500MB of disk space in /tmp/ provided to the function, so it would need to contain the unzipped and the zipped file. This will likely cause diffculties if the files total more than this disk space.

Upvotes: 3

Related Questions