CODEWITHSUNDEEP
c#amazon-web-services
kunal
kunal

Reputation: 966

multipart upload error in c#

I am trying to upload a file using AWS when i set the part size as 5MB the code works fine, but when i try to change the part size to 1MB it gives me an exception: The code is

            string strusername = "user1";       
            strlocalpath = "C:\\file1.zip";    
            string BUCKET_NAME = "bucket1";
            string filename = "file1.zip"
            string keypath = strusername + "/" + filename;
            string keyName = "123";
            string filePath = strlocalpath;
            // List to store upload part responses.
            List<UploadPartResponse> uploadResponses = new List<UploadPartResponse>();

            // 1. Initialize.
            InitiateMultipartUploadRequest initRequest =
                new InitiateMultipartUploadRequest()
                .WithBucketName(BUCKET_NAME)
                .WithKey(keyName);

            InitiateMultipartUploadResponse initResponse =
                s3Client.InitiateMultipartUpload(initRequest);

            // 2. Upload Parts.
            long contentLength = new FileInfo(filePath).Length;
            //Set Part size 
            long partSize = 1*1024*1024; // 5 MB

            try
            {
                long filePosition = 0;
                for (int i = 1; filePosition < contentLength; i++)
                {

                    if (filePosition + partSize > contentLength)
                    {
                        partSize = contentLength - filePosition;
                    }
                    // Create request to upload a part.
                    UploadPartRequest uploadRequest = new UploadPartRequest()
                        .WithBucketName(BUCKET_NAME)
                        .WithKey(keyName)
                        .WithUploadId(initResponse.UploadId)
                        .WithPartNumber(i)
                        .WithPartSize(partSize)
                        .WithFilePosition(filePosition)
                        .WithFilePath(filePath)
                        .WithTimeout(60*60*60);

                    // Upload part and add response to our list.
                    uploadResponses.Add(s3Client.UploadPart(uploadRequest));

                    filePosition += partSize;



                    Console.WriteLine("\nTotal uploaded size = " + filePosition.ToString());
                }

                // Step 3: complete.
                CompleteMultipartUploadRequest compRequest =
                    new CompleteMultipartUploadRequest()
                    .WithBucketName(BUCKET_NAME)
                    .WithKey(keyName)
                    .WithUploadId(initResponse.UploadId)
                    .WithPartETags(uploadResponses);

                CompleteMultipartUploadResponse completeUploadResponse =
                    s3Client.CompleteMultipartUpload(compRequest);

            }
            catch (Exception exception)
            {
                Console.WriteLine("Exception occurred: {0}", exception.Message);
                s3Client.AbortMultipartUpload(new AbortMultipartUploadRequest()
                    .WithBucketName(BUCKET_NAME)
                    .WithKey(keyName)
                    .WithUploadId(initResponse.UploadId));
            }

The exception is:

<Error>
<Code>EntityTooSmall</Code>
<Message>Your proposed upload is smaller than the minimum allowed size</Message>   
<ETag>d9c00192bcf6bf7412814a8fe0422b0c</ETag>
<MinSizeAllowed>5242880</MinSizeAllowed>
<ProposedSize>1048576</ProposedSize>
<RequestId>PBG04E031C012F34</RequestId>
<HostId>VEjvpkjuk89yS4xW6Bl/+NPpb3yxvbbe7ijjPmTrlXc7hnjj89kjkm</HostId> 
<PartNumber>1</PartNumber></Error>

I want to upload the file in chunks of 1 MB or 2MB is it possible to do that?

Thanks

Upvotes: 1

Views: 2972

Answers (3)

Farshid Ashouri
Farshid Ashouri

Reputation: 17691

There is a simple rule: If your file is less than 5Mb, then upload it in one part.

Upvotes: 0

bland
bland

Reputation: 1994

Looks like you got your sizing wrong.

S3 Documentation: Upload Part

Each part must be at least 5 MB in size, except the last part. There is no size limit on the last part of your multipart upload.

So no, you cannot specify to do parts less than 5MB. When doing multipart, it will automatically allow the last part, assuming the last part is not the first/only part, to be less than 5MB.

Glacier Documentation: Amazon's Initiate MultiPart Upload documentation states:

When you initiate a multipart upload, you specify the part size in number of bytes. The part size must be a megabyte (1024 KB) multiplied by a power of 2—for example, 1048576 (1 MB), 2097152 (2 MB), 4194304 (4 MB), 8388608 (8 MB), and so on. The minimum allowable part size is 1 MB, and the maximum is 4 GB.

Your code has long partSize = 1*1024*1024 which is 1048576 which is 1 MB from the above, not 5MB as your comment states.

And ofnote for future reference or same issue, from the same link:

Every part you upload using this upload ID, except the last one, must have the same size. The last one can be the same size or smaller. For example, suppose you want to upload a 16.2 MB file. If you initiate the multipart upload with a part size of 4 MB, you will upload four parts of 4 MB each and one part of 0.2 MB.

Upvotes: 2

Michael - sqlbot
Michael - sqlbot

Reputation: 179064

The minimum part size for S3 uploads is 5 MB.

http://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html

Obviously, the last part will usually be smaller, which is fine.

A side effect of the fact that the "last part" can be smaller is that if you have, for example, a 500 KB file and you send it as a multipart upload, the "first part" is also the "last part" and this still works, because it satisfies the rule that the "last part" can be smaller than 5MB, but you still have to expressly use a part-size not smaller than 5 MB.

I discovered this while testing a multipart-uploaded that defaults to 64MB. It still works fine on smaller files, which get multipart-uploaded as a single "part" which is less than 5MB.

Upvotes: 3

Related Questions