CODEWITHSUNDEEP
javaandroidiosamazon-web-servicesamazon-s3
Kamchatka
Kamchatka

Reputation: 3675

How to upload a file to S3 only if not uploaded already?

I'm using the AWS Transfer Manager to backup a lot of files to S3. Sometimes the backup fails in the middle in the middle, and I don't want to re-upload all the files, but only the ones that haven't been uploaded yet.

Is there something baked in the Transfer Manager or the S3 Put Request that would let me do that automatically, or is my only solution to check the MD5 of the file with a HEAD request first, and see if it's different before starting the upload.

Thanks!

Upvotes: 1

Views: 1314

Answers (2)

John Rotenstein
John Rotenstein

Reputation: 270164

Rather than coding your own solution, you could use the AWS Command-Line Interface (CLI) to copy or sync the files to Amazon S3.

For example:

aws s3 sync <directory> s3://my-bucket/<directory>

The sync command will only copy files that are not in the destination. So, just run it on a regular basis and it will copy all the files to the S3 bucket!

Upvotes: 1

Rohan Dubal
Rohan Dubal

Reputation: 847

You can do that using the continue with block. For every upload you can define a retry strategy in the failed upload case. For e.g.:

    [[transferManager upload:uploadRequest] continueWithBlock:^id(AWSTask *task) {
        if (task.error){
            // Handle failed upload here
        }

        if (task.result) {
            //File uploaded successfully.
        }
        return nil;
    }];

You could also create a list of tasks and then use

NSMutableArray *tasks = [NSMutableArray new];    
AWSTask * taskForUpload = [transferManager upload:uploadRequest];
[tasks addObject:taskForUpload];
// add more tasks as required    
[[AWSTask taskForCompletionOfAllTasks:tasks] continueWithBlock:^id(AWSTask *task) {
                        if (task.error != nil) {
                            // Handler error / failed uploads here
                        } else {
                            // Handle successful uploads here
                        }                        
                        return nil;
                    }];

This will perform all the tasks in the list and then give you list of errors which you can retry.

Thanks, Rohan

Upvotes: 0

Related Questions