CODEWITHSUNDEEP
ruby-on-railsrubyruby-on-rails-3amazon-s3amazon-web-services
99miles
99miles

Reputation: 11212

Stream uploading large files using aws-sdk

Is there a way to stream upload large files to S3 using aws-sdk?

I can't seem to figure it out but I'm assuming there's a way. Thanks

Upvotes: 3

Views: 5276

Answers (2)

Steffen Opel
Steffen Opel

Reputation: 64711

Update

My memory failed me and I didn't read the quote mentioned in my initial answer correctly (see below), as revealed by the API documentation for (S3Object, ObjectVersion) write(data, options = {}) :

Writes data to the object in S3. This method will attempt to intelligently choose between uploading in one request and using #multipart_upload.

[...] You can pass :data or :file as the first argument or as options. [emphasis mine]

The data parameter is the one to be used for streaming, apparently:

:data (Object) — The data to upload. Valid values include:

[...] Any object responding to read and eof?; the object must support the following access methods:

read                     # all at once
read(length) until eof?  # in chunks

If you specify data this way, you must also include the :content_length option.

[...]

:content_length (Integer) — If provided, this option must match the total number of bytes written to S3 during the operation. This option is required if :data is an IO-like object without a size method.

[emphasis mine]

The resulting sample fragment might look like so accordingly:

# Upload a file.
key = File.basename(file_name)
s3.buckets[bucket_name].objects[key].write(:data => File.open(file_name), 
    :content_length => File.size(file_name))
puts "Uploading file #{file_name} to bucket #{bucket_name}."

Please note that I still haven't actually tested this, so beware ;)

Initial Answer

This is explained in Upload an Object Using the AWS SDK for Ruby:

Uploading Objects

  1. Create an instance of the AWS::S3 class by providing your AWS credentials.
  2. Use the AWS::S3::S3Object#write method which takes a data parameter and options hash which allow you to upload data from a file, or a stream. [emphasis mine]

The page contains a complete example as well, which uses a file rather than a stream though, the relevant fragment:

# Upload a file.
key = File.basename(file_name)
s3.buckets[bucket_name].objects[key].write(:file => file_name)
puts "Uploading file #{file_name} to bucket #{bucket_name}."

That should be easy to adjust to use a stream instead (if I recall correctly you might just need to replace the file_name parameter with open(file_name) - make sure to verify this though), e.g.:

# Upload a file.
key = File.basename(file_name)
s3.buckets[bucket_name].objects[key].write(:file => open(file_name))
puts "Uploading file #{file_name} to bucket #{bucket_name}."

Upvotes: 6

Tom Andersen
Tom Andersen

Reputation: 7200

I don't know how big the files you want to upload are, but for large files a 'pre-signed post' allows the user operating the browser to bypass your server and upload directly to S3. That may be what you need - to free up your server during an upload.

Upvotes: 1

Related Questions