CODEWITHSUNDEEP
rubylinuxamazon-s3
Tom
Tom

Reputation: 19304

S3: my multi-part uploader doesn't work for files > 5GB

I'm trying to upload many large (30GB+) files to an S3 bucket using the multipart upload feature.

Here is the code. I've put it in a gist to avoid cluttering this page: https://gist.github.com/twslankard/5296081

Files smaller than 5GB are uploaded with no problems. Files larger than 5GB seem to get "clipped" at 5GB, according to the sizes of the objects in the S3 management console.

During the upload, I get no exceptions or error messages of any kind. I also find nothing in the documentation to suggest I'm doing something wrong.

Can anyone see what I'm doing wrong?

EDIT: I've learned from AWS support that multipart upload is not yet supported in the Ruby SDK

Upvotes: 2

Views: 1813

Answers (2)

Tom
Tom

Reputation: 19304

According to AWS, multipart upload is not supported in the Ruby SDK.

However, I've found a python boto-based multipart uploading script.

https://gist.github.com/chrishamant/1556484

EDIT: Ruby SDK now supports multipart upload http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu-ruby-sdk.html

Upvotes: 1

Arman H
Arman H

Reputation: 5618

S3 has a policy document that authorizes the form and imposes limits on the files that can be uploaded. One of the limits is content length. Verbatim from Amazon:

Content length rule, which checks that the size of an uploaded file is between a given minimum and maximum value. If this rule is not included in a policy document, users will be able to upload files of any size up to the 5GB limit imposed by S3.

Upvotes: 3

Related Questions