CODEWITHSUNDEEP
amazon-web-servicesamazon-s3
Dima Svider
Dima Svider

Reputation: 450

How to combine multiple S3 objects in the target S3 object w/o leaving S3

I understand that the minimum part size for uploading to an S3 bucket is 5MB Is there any way to have this changed on a per-bucket basis?

The reason I'm asking is there is a list of raw objects in S3 which we want to combine in the single object in S3.

Using PUT part/copy we are able to "glue" objects in the single one providing that all objects except last one are >= 5MB. However sometimes our raw objects are not big enough and in this case when we try to complete multipart uploading we're getting famous error "Your proposed upload is smaller than the minimum allowed size" from AWS S3.

Any other idea how we could combine S3 objects without downloading them first?

Upvotes: 2

Views: 2171

Answers (2)

wwadge
wwadge

Reputation: 3544

"However sometimes our raw objects are not big enough... "

You can have a 5MB garbage object sitting on S3 and do concatenation with it where part 1 = 5MB garbage object, part 2 = your file that you want to concatenate. Keep repeating this for each fragment and finally use the range copy to strip out the 5MB garbage

Upvotes: 1

Renato Gama
Renato Gama

Reputation: 16519

There is no way to have the minimum part size changed

You may want to either;

  1. Stream them together to AWS (which does not seem like an option, otherwise you would already be doing this)
  2. Pad the file so it fill the minimum size of 5MB (what can or cannot be feasible to you since this will increase your bill). You will have the option to either use infrequent access (when you access these files rarely) or reduced redundancy (when you can recover lost files) if you think it can be applied to these specific files in order to reduce the impact.
  3. Use an external service that will zip (or "glue" them together) your files and then re-upload to S3. I dont know if such service exists, but I am pretty sure you can implement it your self using a lambda function (I have even tried something like this in the past; https://github.com/gammasoft/zipper-lambda)

Upvotes: 0

Related Questions