CODEWITHSUNDEEP
javaamazon-web-servicesamazon-s3
Sridhar Voorakkara
Sridhar Voorakkara

Reputation: 313

Error with AWS Java SDK doing an upload with metadata

I am trying to upload a file to a S3 container and before doing the upload, I am setting the metadata of the file. The upload fails with an error saying signature doesn't match. Below is the code I am using :

public URL send(File f, HashMap<String,String> metadata, String type) throws Exception {
    String path = type+"/"+f.getName();

    InitiateMultipartUploadRequest req = new InitiateMultipartUploadRequest(container, secretKey).withKey(path);
    req.setCannedACL(CannedAccessControlList.AuthenticatedRead);

    if (metadata != null) {
        ObjectMetadata objectMetadata = new ObjectMetadata();
        Set<String> keys = metadata.keySet();
        Iterator<String> i = keys.iterator();
        while (i.hasNext()) {
            String key = i.next();
            objectMetadata.addUserMetadata(key, metadata.get(key));
        }
        req.setObjectMetadata(objectMetadata);
    }


    InitiateMultipartUploadResult res = s3client.initiateMultipartUpload(req);

    String uploadId = res.getUploadId();
    long fileSize = f.length();
    //check the size doesn't exceed max limit
    if (fileSize > MAX_OBJ_SIZE) {
        throw new Exception("Object size exceeds repository limit");
    }
    long chunkSize = 1024 * 1024 * 16;
    int chunks = (int) (fileSize/chunkSize + 2);
    List<PartETag> chunkList = new ArrayList<PartETag>();
    long pos = 0; 
    try {
        for (int i = 1; i < chunks; i++) {

            if ((chunks -i) < 2) {
                chunkSize = fileSize - pos;
            }

            UploadPartRequest upReq = new UploadPartRequest()
                    .withBucketName(container).withKey(path)
                    .withUploadId(uploadId).withPartNumber(i)
                    .withFileOffset(pos).withFile(f)
                    .withPartSize(chunkSize);

            PartETag pTag = null;
            // repeat the upload until it succeeds.
            boolean repeat;  
            do {
                repeat = false;  // reset switch
                try {
                    // Upload part and add response to our list.
                    pTag =   s3client.uploadPart(upReq).getPartETag(); 
                } 
                catch (Exception ex) {
                    repeat = true; // repeat
                }
            } while (repeat);

            chunkList.add(pTag);
            pos = pos + chunkSize;
        }
        CompleteMultipartUploadRequest compl = new CompleteMultipartUploadRequest(
                container, secretKey, uploadId, chunkList).withKey(path);
        CompleteMultipartUploadResult  complRes = s3client.completeMultipartUpload(compl);
        return new URL(URLDecoder.decode(complRes.getLocation(), "UTF-8"));
    }
    catch (Exception ex) {
        s3client.abortMultipartUpload(new AbortMultipartUploadRequest(container, 
                secretKey, uploadId));
        throw new Exception("File upload error: "+ex.toString());
    }
}

Below is the error I am getting :

 com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 0805716BBD0662AB, AWS Error Code: SignatureDoesNotMatch, AWS Error Message: The request signature we calculated does not match the signature you provided. Check your key and signing method., S3 Extended Request ID: wNAzUyrLZgWCazZFe3KpMHO0uh0FM5FF7fiwBzN1A2YDEYS5hKZBYh5nWSjIhnhG
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:767)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:414)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:228)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3316)
at com.amazonaws.services.s3.AmazonS3Client.initiateMultipartUpload(AmazonS3Client.java:2401)
at net.timbusproject.storage.awss3.S3Client.send(S3Client.java:134)

Line 134 in S3Client.java where the error is occurring is :

 InitiateMultipartUploadResult res = s3client.initiateMultipartUpload(req);

The upload works fine if I am not attaching any metadata. i.e, if I comment the below line, the upload works :

 req.setObjectMetadata(objectMetadata);

I am unable to figure out why the request fails when metadata is set. Am I missing any step in the upload process ?

Upvotes: 4

Views: 1963

Answers (1)

Sridhar Voorakkara
Sridhar Voorakkara

Reputation: 313

I was able to work around this problem by URL encoding the metadata keys and values. 

objectMetadata.addUserMetadata(URLEncoder.encode(key, "UTF-8"), URLEncoder.encode(metadata.get(key),"UTF-8"));

Obviously the metadata seems to have some offending characters which are messing with the AWS calls. This workaround will let upload complete without error and also updates the metadata but the strings remain url encoded, which can be a problem later.

Upvotes: 5

Related Questions