Serving large files from S3 on Rails

Question

I need to be able to serve a large file as a download from a Rails app, from an S3 bucket. I would typically just give the user the URL to the 3S bucket, but I need it to remain masked so I can expire or remove the URL at will. I'm doing something like the following:

def download
  purchase = Purchase.find(params[:id])

  data = open(purchase[:download_url])
  send_data data.read, filename: purchase[:file_name]
end

It works for smaller files but on one that's, say 1GB, because of Heroku's timeout limits, the user gets a big fat 520 error.

I understand what's going on here: the entire file is being opened up in my application before being sent over to the user, and that's why for larger files it times out. What I'm wondering is if there's a way around this without giving the user S3 URL? Is it possible to, rather than download the file, just mask the URL (not redirect)?

Any help is appreciated!

Edit

Based on other answers on SO, I tried to implement an expiring URL:

def download
  purchase = Purchase.find(params[:id])
  path = purchase[:download_url].sub! 'entire URL until bucket path', ''
  s3URL = AWS::S3::S3Object.new(S3_BUCKET, path)

  redirect_to s3URL.url_for(:read).to_s
end

That just redirects to an AccessDenied on AWS, but maybe I'm building the URL incorrectly. Again, any help is appreciated!

Answer 1

You don't need to send the data, just set up a proxy url, that redirects to the S3 url, or whatever else you need to do programatically.

Also bear in mind that S3 urls have expiration baked in.

Answer 2

I think your S3 object url is inaccessible because the bucket/object doesn't have public-read.

To allow people direct download from your S3 bucket, you can either:

• Make your bucket public accessible, or
• Create a presigned url to authorize anonymous S3 users download the file.

Make S3 bucket public accessible

If you follow this method, you can just give the S3 object url as in your updated question. You just need to configure the bucket in AWS.

1. Go to S3 console.
2. Open your bucket properties (right click your bucket and click Properties).
3. Click on Permission section, and then click Add bucket policy.

4. Copy and paste the following bucket policy:

   {
     "Version":"2012-10-17",
     "Statement":[
       {
         "Sid":"AddPerm",
         "Effect":"Allow",
         "Principal": "*",
         "Action":["s3:GetObject"],
         "Resource":["arn:aws:s3:::examplebucket/*"]
       }
     ]
   }
   

   Source: http://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-2

5. Click Save. Now, any users can download any objects inside this bucket as long as they have the URL. As a tip, to disallow guessing the URL, you can rename your object key with reasonable length random string.

Create a presigned url

In this method, only user who has the presigned url can download your S3 object in a certain period (by default, it's one week). After that, he/she need to get a new one. To create presigned S3 object:

def download
  purchase = Purchase.find(params[:id])
  path = purchase[:download_url].sub! 'entire URL until bucket path', ''

  s3 = Aws::S3::Resource.new(region:'us-west-2')
  object = s3.bucket(S3_BUCKET).object(path)
  redirect_to object.presigned_url(:get)
end

You may also set some options in the object.presigned_url method like:

object.presigned_url(:get, expires_in: 3600, response_content_disposition: 'attachment; filename=original_file_name.zip')

Read more in http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Object.html#presigned_url-instance_method.

You need to choose which one is suitable for your case. For non-sensitive and public S3 object, I prefer to make S3 bucket public accessible. But, I think, for your case (purchase something and then download), it's better to implement the presigned URL.