CODEWITHSUNDEEP
amazon-web-servicesgoamazon-s3
serverhorror
serverhorror

Reputation: 816

Copy S3 Object with MultiPartUpload

I need to rename a quite a bunch of objects in AWS S3. For small objects the following snippet works flawlessly:

input := &s3.CopyObjectInput{
    Bucket:     aws.String(bucket),
    Key:        aws.String(targetPrefix),
    CopySource: aws.String(source),
}
_, err = svc.CopyObject(input)
if err != nil {
    panic(errors.Wrap(err, "error copying object"))
}

I am running into the S3 size limitation for larger objects. I understand I need to copy the object using a multi part upload. This is what I tried so far:

multiPartUpload, err := svc.CreateMultipartUpload(
    &s3.CreateMultipartUploadInput{
        Bucket: aws.String(bucket),
        Key: aws.String(targetPrefix), // targetPrefix is the new name
    },
)
if err != nil {
    panic(errors.Wrap(err, "could not create MultiPartUpload"))
}
resp, err := svc.UploadPartCopy(
    &s3.UploadPartCopyInput{
        UploadId:   multiPartUpload.UploadId,
        Bucket:     aws.String(bucket),
        Key:        aws.String(targetPrefix),
        CopySource: aws.String(source),
        PartNumber: aws.Int64(1),
    },
)
if err != nil {
    panic(errors.Wrap(err, "error copying multipart object"))
}
log.Printf("copied: %v", resp)

The golang SDK bails out on me with:

InvalidRequest: The specified copy source is larger than the maximum allowable size for a copy source: 5368709120

I have also tried the following approach but I do not get any parts listed here:

multiPartUpload, err := svc.CreateMultipartUpload(
    &s3.CreateMultipartUploadInput{
        Bucket: aws.String(bucket),
        Key: aws.String(targetPrefix), // targetPrefix is the new name
    },
)
if err != nil {
    panic(errors.Wrap(err, "could not create MultiPartUpload"))
}
err = svc.ListPartsPages(
    &s3.ListPartsInput{
        Bucket:   aws.String(bucket),       // Required
        Key:      obj.Key,                  // Required
        UploadId: multiPartUpload.UploadId, // Required
    },
    // Iterate over all parts in the `CopySource` object
    func(parts *s3.ListPartsOutput, lastPage bool) bool {
        log.Printf("parts:\n%v\n%v", parts, parts.Parts)
        // parts.Parts is an empty slice
        for _, part := range parts.Parts {
            log.Printf("copying %v part %v", source, part.PartNumber)
            resp, err := svc.UploadPartCopy(
                &s3.UploadPartCopyInput{
                    UploadId:   multiPartUpload.UploadId,
                    Bucket:     aws.String(bucket),
                    Key:        aws.String(targetPrefix),
                    CopySource: aws.String(source),
                    PartNumber: part.PartNumber,
                },
            )
            if err != nil {
                panic(errors.Wrap(err, "error copying object"))
            }
            log.Printf("copied: %v", resp)
        }
        return true
    },
)
if err != nil {
    panic(errors.Wrap(err, "something went wrong with ListPartsPages!"))
}

What am I doing wrong or am I missunderstanding something?

Upvotes: 1

Views: 1811

Answers (1)

Alex Guerra
Alex Guerra

Reputation: 2736

I think that ListPartsPages is the wrong direction because it works on "Multipart Uploads" which is a different entity than an an s3 "Object". So you're listing the already-uploaded parts to the multipart upload you just created.

Your first example is close to what's needed, but you need to manually split the original file into parts, with the range of each part specified by UploadPartCopyInput's CopySourceRange. At least that's my take from reading the documentation.

Upvotes: 2

Related Questions