CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-s3
niks_4143
niks_4143

Reputation: 97

How to upload multiple files and combine them in S3

I have a large size exe file (> 100 MB) which I need to upload to S3. Now since I have a reliable network, I want to make this process faster by dividing the file into chunks and then uploading these chunks in parallel. But I don't know how to merge these chunks back together to get the original exe?

Although python provides a way to upload a file into chunks but those chunks are uploaded serially. Is there any way to upload these chunks in parallel and then merge them back together?

Upvotes: 6

Views: 6824

Answers (1)

Mircea
Mircea

Reputation: 10566

S3 already has support for this.

It's called a multipart upload: http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html

Use a client (for example s3cmd http://s3tools.org/kb/item13.htm) that knows how to do this or use do it yourself through the SDK and your favorite language.

Upvotes: 5

Related Questions