CODEWITHSUNDEEP
asp.net.netasynchronousamazon-s3multipart
Santhu Santhosh
Santhu Santhosh

Reputation: 11

Asynchronous multipart uploads to Amazon S3 with ASP.NET

I am able to initiate asynchronous uploads to S3, however they are somehow not ending up as a file inside my S3 bucket and I see an error 'WithPartETags cannot be empty'. Here is the complete code

InitiateMultipartUploadRequest initRequest =
    new InitiateMultipartUploadRequest()
    .WithBucketName(existingBucketName)
    .WithKey(Path.Combine(S3Path + "/", finfo.Name));

InitiateMultipartUploadResponse initResponse =
    s3Client.InitiateMultipartUpload(initRequest);

// 2. Upload Parts.
long contentLength = finfo.Length;
long partSize = 15728640;//52428800-50MB 104857600- 100 MB - 5242880 - 5 MB
try
{
    long filePosition = 0;
    for (int i = 1; filePosition < contentLength; i++)
    {

        // Create request to upload a part.
        UploadPartRequest uploadRequest = new UploadPartRequest()
            .WithBucketName(existingBucketName)
            .WithKey(Path.Combine(S3Path + "/", finfo.Name))
            .WithUploadId(initResponse.UploadId)
            .WithPartNumber(i)
            .WithPartSize(partSize)
            .WithFilePosition(filePosition)
            .WithFilePath(finfo.FullName);

        // Upload part and add response to our list.
        //uploadResponses.Add(s3Client.UploadPart(uploadRequest));

        IAsyncResult ar = s3Client.BeginUploadPart(uploadRequest, null, null);
        ListObj.Add(new ThreadList() { _iasyncResult = ar });

        filePosition += partSize;
        Console.WriteLine("Length Written - " + filePosition + " .Content Length - " + contentLength);
    }

    bool uploadsComplete = false;
    while (!uploadsComplete)
    {
        bool individualuploadscomplete = true;
        foreach (var obj in ListObj)
        {
            if (!obj._iasyncResult.IsCompleted)
            {
                individualuploadscomplete = false;
                break;
            }

        }
        if (individualuploadscomplete)
        {
            uploadsComplete = true;
        }
    }

    foreach (var obj in ListObj)
    {
        s3Client.EndUploadPart(obj._iasyncResult);
    }

    //// Step 3: complete.
    CompleteMultipartUploadRequest compRequest =
        new CompleteMultipartUploadRequest()
        .WithBucketName(existingBucketName)
        .WithKey(Path.Combine(S3Path + "/", finfo.Name))
        .WithUploadId(initResponse.UploadId);
        //.WithPartETags(uploadResponses);

    CompleteMultipartUploadResponse completeUploadResponse =
        s3Client.CompleteMultipartUpload(compRequest);

Upvotes: 1

Views: 1774

Answers (1)

Norm Johanson
Norm Johanson

Reputation: 3177

Not sure why you have the setting of the PartETags commented out for the complete multipart upload call but you need to add that code back in. Also when you are calling the EndUploadPart method you need to capture that UploadResponse that comes back from that.

You also might want to look into the TransferUtility found in the Amazon.S3.Transfer namespace. Its upload methods are designed to handle what you are attempting to accomplish for large objects, see Using the High-Level .NET API for Multipart Upload for details and example snippets.

Upvotes: 2

Related Questions