CODEWITHSUNDEEP
ruby-on-railsamazon-s3
Tarang
Tarang

Reputation: 75945

Ruby S3 Multipart pause and resume (add a part after calling close)

I'm using the aws-sdk gem and I'm trying to basically upload a really large file (it takes 2 days).

The file is uploaded in chunks but sometimes the script will crash and I would like to resume uploading (the next chunk).

During uploading we want to close close the multipart upload (so we can access the s3 data that has been uploaded so far).

Is it possible to add a part after the multipart upload is closed? (say the next day) basically resuming the upload?

Upvotes: 0

Views: 1108

Answers (1)

Steffen Opel
Steffen Opel

Reputation: 64701

Is it possible to add a part after the multipart upload is closed? (say the next day) basically resuming the upload?

Not as such, but you can simulate the affect you desire.

Background

Uploading Objects Using Multipart Upload API allows you to upload a single object as a set of parts:

Each part is a contiguous portion of the object's data. You can upload these object parts independently and in any order. If transmission of any part fails, you can retransmit that part without affecting other parts. After all parts of your object are uploaded, Amazon S3 assembles these parts and creates the object. [emphasis mine]

This is further detailed in Complete Multipart Upload:

You first initiate the multipart upload and then upload all parts using the Upload Parts operation (see Upload Part). [...] Upon receiving this request, Amazon S3 concatenates all the parts in ascending order by part number to create a new object. [...] You must ensure the parts list is complete, this operation concatenates the parts you provide in the list. [...] [emphasis mine]

That is, the upload operation is finalized here and cannot be resumed by uploading another part. (Technically speaking the upload ID required for any operation on an initiated multipart upload is not available/valid anymore).

Solution

You can simple initiate a new multipart upload and upload your previously uploaded S3 object as the first part of this new multipart object by means of the Upload Part - Copy operation, which Uploads a part by copying data from an existing object as data source.

Upvotes: 2

Related Questions