CODEWITHSUNDEEP
amazon-web-servicesamazon-cloudfront
Naftuli Kay
Naftuli Kay

Reputation: 91630

Uploads to S3 through CloudFront via Signed URLs?

I have CloudFront using an S3 origin. I'd like to be able to generate signed URLs for uploading values into S3, if possible through the CloudFront distribution.

I've been poring over the docs but I haven't found anything on this. I'd imagine that as long as the query parameters are passed along, but is this the case?

Not a duplicate of this: I'm using signed URLs.

Upvotes: 10

Views: 9731

Answers (3)

Oleg
Oleg

Reputation: 24988

Yes, you can use a CloudFront distribution to ingest S3 file uploads.

Depending on your setup, S3 Transfer Acceleration suggested in another answer may still be a better option, it's certainly better documented and has more implementation examples!

Create a new CF distribution

  • Use target S3 bucket as origin
  • Configure origin S3 bucket access to use a CloudFront origin access identity. While you can automatically update the bucket policy (with s3:GetObject access) as a starting point, we'll be adding/changing that to s3:PutObject anyway.

Configure S3 bucket access to use a CloudFront origin access identity

  • Configure "Allowed HTTP methods" to include POST

Configure allowed HTTP methods

  • We'll be allowing writing to the bucket, so it would be a good idea to "Restrict viewer access" and require CloudFront signed URLs or signed cookies for access.

Require signed URLs or cookies

Configure S3 bucket policy

  • If you've started with an automatically updated bucket policy above, your bucket policy should contain something like below, where XXXXX is the OAI ID and YOUR-TARGET-BUCKET is, well, your target bucket:
    "Version": "2008-10-17",
    "Id": "PolicyForCloudFrontPrivateContent",
    "Statement": [
        {
            "Sid": "1",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity XXXXX"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::YOUR-TARGET-BUCKET/*"
        }
    ]
}
  • You can add s3:PutObject as an allowed action, but if you're setting up the distribution for content ingestion only (i.e. delivered through another distribution, or not publicly available at all) you could just replace the allowed action:
            "Action": "s3:PutObject",

Upload files to S3 using the CloudFront distribution

Once the distribution is updated, you would be able to add files to your bucket by making a PUT request to https://your-distribution-url/desired-s3-key-name using Postman or something. Remember to use a signed url here if that's what you've configured.

Your client connections to the CF edge should be consistently faster, while anecdotally S3 acceleration speed comparison isn't always favorable.

If I understand AWS pricing correctly, you will be charged for "Regional Data Transfer Out to Origin" ($0.02-$0.16/GB depending on region) compared with S3 Transfer Acceleration charges ($0.04-$0.08/GB depending on region.)

Upvotes: 4

Michael - sqlbot
Michael - sqlbot

Reputation: 179084

I'd imagine that as long as the query parameters are passed along, but is this the case?

Not quite. CloudFront can be configured to pass through the query string parameters, but it still won't support this use case, because CloudFront injects an X-Amz-Cf-Id header into the request that it sends to S3. Because the header starts with X-Amz-*, S3 requires it to be signed... but this is impossible, since you have no way of knowing its value.

A CloudFront origin access identity can be given permission to write to a bucket, but this is also messy, because then any path CloudFront can read from can also be written to -- CloudFront signed URLs allow "access" -- whatever kind of access CloudFront itself has -- and can't be configured as read-only, if CloudFront, via the OAI, is allowed to write to the bucket.

Transfer Acceleration is a better solution since it uses the CloudFront network for transport, but understands native S3 signed URLs. Your argument related to wanting to use the domain name rather than the bucket name in the Lambda function signing the URLs breaks down... because you can't sign an S3 URL in any event, without actually knowing the bucket name.

Upvotes: 3

Sébastien Stormacq
Sébastien Stormacq

Reputation: 14905

Is the reason is to provide low latency access to user uploading files to a central bucket, from all around the world ? In that case, I would recommend using S3 Transfer Acceleration instead.

You might want to use Transfer Acceleration on a bucket for various reasons, including the following:

  • You have customers that upload to a centralized bucket from all over the world.

  • You transfer gigabytes to terabytes of data on a regular basis across continents.

  • You are unable to utilize all of your available bandwidth over the Internet when uploading to Amazon S3.

https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html

Amazon S3 transfer acceleration is using Cloudfront under the scene.

Transfer Acceleration takes advantage of Amazon CloudFront’s globally distributed edge locations. As the data arrives at an edge location, data is routed to Amazon S3 over an optimized network path.

Upvotes: 5

Related Questions