CODEWITHSUNDEEP
c#.netamazon-web-servicesfile-uploadamazon-s3
Mike Upjohn
Mike Upjohn

Reputation: 1297

Amazon S3 - Re-starting a failed upload

I am using Amazon S3 with the .NET SDK, and the low-level API that they provide. I am looking for advice on ways to try and restart a multi part upload of a file. That's a restart as in, for example the network connection drops out, or there is an internal exception which terminates the program before all parts of the file are uploaded to Amazon.

I am using their example code as per the documentation for multipart upload:-

 for (int i = filePartCounts[0]; filePosition < contentLength; i++)
 {
     if (i == 10)
     {
         throw new Exception();
     }

     UploadPartRequest uploadRequest = new UploadPartRequest
     {
         BucketName = "MyAwesomeBucket",
         Key = "my-file-" + now + ".mts",
         UploadId = initiateResponse.UploadId,
         PartNumber = i,
         PartSize = partSize,
         FilePosition = filePosition,
         FilePath = "C:\\path\\file.MTS",
         Timeout = new TimeSpan(0, 15, 00)
      };

      uploadResposes.Add(client.UploadPart(uploadRequest));
      File.WriteAllText(@"C:\\file\\data.txt", "my-file-" + now + ".mts/" + i + "/" + maxPartNumber);

      filePosition += partSize;

      Console.WriteLine("Successfull uploaded part " + i + " of " + maxPartNumber + " - Bytes " + (filePosition - partSize) + " bytes to " + filePosition + " bytes");
}

As this is a prototype, I am saving a record to a text file, with the KeyName, Last Successful Part Number and Maximum Number of Part Numbers in this file, so that I can read this next time the program runs, get the key name, and start at the first chunk that didn't upload. For now, I've simulated the program erroring by manually calling:-

if (i == 10)
{
    throw new Exception();
}

so the program exits after 9 of 73 chunks.

I face an issue when trying to get the item back from the Amazon S3 servers. I get an exception when I call:-

client.GetObject(request);

I want to get back the parts of the file uploaded before continuing with the rest of the parts. For obvious reasons I get an exception of 'The specified key does not exist.' because the file is not yet completed on Amazon S3.

My question is, if this does not work, how do I get back the parts of the file that did successfully upload to Amazon S3, to then continue adding parts 10 onwards to it?

Any help is massively appreciated! Thanks.

Upvotes: 0

Views: 1883

Answers (1)

Matt Houser
Matt Houser

Reputation: 36113

A multi-part upload will only appear as an object in the bucket after CompleteMultipartUpload is called.

http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html

Before then, you have two choices:

Option 1: You can abort the existing multi-part upload using AbortMultipartUpload and start again.

http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadAbort.html

Option 2: You can use ListParts to get the current list of uploaded parts, then upload the remainder of the parts.

http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadListParts.html

Upvotes: 2

Related Questions