CODEWITHSUNDEEP
amazon-s3pre-signed-url
silkAdmin
silkAdmin

Reputation: 4810

Adding Content-Encoding header to signed URL uploaded files

I would need to upload Gziped content to S3 via a signed URL.

Here is how I generate the signed URL with a JS backend: 

s3.createPresignedPost({
  Bucket: 'name',
  Fields: {
    key: 'key'
  }
})

I have tried passing the Content-Encoding header to the signedURL POST request but that did not work. The headers are not set properly on the s3 object.

I have also tried setting up a post upload lambda to update the metadata. It failed with an error File is identical error

Finally I have tried using cloudfront + a lambda to force a header. This failed too with an error stating that Content-Enconding is a protected error.

Upvotes: 2

Views: 5395

Answers (1)

Imran
Imran

Reputation: 6235

--Update Start--

For uploading to S3 via Ajax or JS scripts, I would advise to use s3.getSignedUrl method. s3.createPresignedPost is meant for only direct browser uploads.

Below is example of Ajax jQuery Upload I created using this guide.

s3.getSignedUrl('putObject', {
    Bucket: 'bucketName',
    Key: 'sample.jpg.gz',
    // This must match with your ajax contentType parameter
    ContentType: 'binary/octet-stream'
    /* then add all the rest of your parameters to AWS puttObect here */
  }, function (err, url) {
    console.log('The URL is', url);
  });

Ajax PUT Script - Take the Url from above function call and use it below.

$.ajax({
    type: 'PUT',
    url: "https://s3.amazonaws.com/bucketName/sample.jpg.gz?AWSAccessKeyId=AKIAIOSFODNN7EXAMPLE&Content-Type=binary%2Foctet-stream&Expires=1547056786&Signature=KyTXdr8so2C8WUmN0Owk%2FVLw6R0%3D",
    //Even thought Content-Encoding header was not specified in signature, it uploads fine.
    headers: {
        'Content-Encoding': 'gzip'
    },
    // Content type must much with the parameter you signed your URL with
    contentType: 'binary/octet-stream',
    // this flag is important, if not set, it will try to send data as a form
    processData: false,
    // the actual file is sent raw
    data: theFormFile
}).success(function () {
    alert('File uploaded');
}).error(function () {
    alert('File NOT uploaded');
    console.log(arguments);
});

In S3 object, you should see Content-Type, Content-Encoding under metadata.

Importent Note When you try to upload via JS scripts which is running on browsers, typically browsers will tend to send OPTIONS method preflight(or CORS check) first before calling PUT method. You will get 403 Forbidden error for OPTIONS since CORS on S3 bucket doesn't allow that. One way, I resolved is by using following CORS configuration on bucket level. Reference

<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>PUT</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>

--Update End--

Did you try like this?. I just tested the policy using sample html given in AWS documentation. Reference - https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTConstructPolicy.html

s3.createPresignedPost({
  Bucket: 'name',
  Conditions: [
    { "Content-Encoding": "gzip" }
  ],
  Fields: {
    key: 'key'
  }
})

Update - Here is my observation so far.

We really need to check your client which is doing upload operation. If you want Content-Encoding set on MetaData, then your Pre-Signed Url should have Content-Encoding property set. If Signed Url doesn't have it but your request header does then it will give you Extra input fields: content-encoding.

I have signed a url with Content-Encoding and uploaded a zipped file with following sample html.

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
  </head>
  <body>
  <form action="http://bucket-name.s3.amazonaws.com" method="post" enctype="multipart/form-data">
    Key to upload: 
    <input type="input"  name="key" value="sample.jpg.gz" /><br />
    Content-Encoding: 
    <input type="input"  name="Content-Encoding" value="gzip" /><br />
    <input type="text"   name="X-Amz-Credential" value="AKIAIOSFODNN7EXAMPLE/20190108/us-east-1/s3/aws4_request" />
    <input type="text"   name="X-Amz-Algorithm" value="AWS4-HMAC-SHA256" />
    <input type="text"   name="X-Amz-Date" value="20190108T220828Z" />
    Tags for File: 
    <input type="hidden" name="Policy" value='bigbase64String' />
    <input type="hidden" name="X-Amz-Signature" value="xxxxxxxx" />
    File: 
    <input type="file"   name="file" /> <br />
    <!-- The elements after this will be ignored -->
    <input type="submit" name="submit" value="Upload to Amazon S3" />
  </form>
</html>

If I do not send Content-Encoding header it gives the error Policy Condition failed: ["eq", "$Content-Encoding", "gzip"]

Note - If you are using https while uploading, please make sure you have proper certificate on S3 endpoint otherwise you will get cert errors.

S3 Screenshot. enter image description here

Upvotes: 4

Related Questions