CODEWITHSUNDEEP
ruby-on-railsajaxruby-on-rails-4amazon-s3
Don P
Don P

Reputation: 63587

Rails: Read the contents from another site

In Rails, how can I make an http request to a page, like "http://google.com" and set the response to a variable?

Basically I'm trying to get the contents of a CSV file off of Amazon S3: https://s3.amazonaws.com/datasets.graf.ly/24.csv

My Rails server needs to return that content as a response to an AJAX request.

  1. Get S3 bucket
  2. Access the file and read it
  3. Render its contents (so the ajax request receives it)

A few questions have suggested screen scraping, but this sounds like overkill (and probably slow) for simply taking a response and pretty much just passing it along.

Upvotes: 1

Views: 156

Answers (2)

iheggie
iheggie

Reputation: 2047

To do as you asked:

Alternatively decode the csv file in rails and pass a json array of arrays back:

off the top of my head it should be something like:

def get_csv
  url = 'http://s3.amazonaws.com/datasets.graf.ly/%d.csv' % params[:id].to_i
  data = open(url).read
  # set header here
  render :text => data
end

Upvotes: 0

Richard Peck
Richard Peck

Reputation: 76774

API

Firstly, you need to know how you're accessing the data

The problems you've cited are only valid if you just access someone's site through HTTP (with something like CURL). As you instinctively know, this is highly inefficient & will likely get your IP blocked for continuous access

A far better way to access data (from any reputable service) is to use their API. This is as true of S3 as Twitter, Facebook, Dropbox, etc:

AWS-SDK

#GemFile
gem "aws-sdk-core", "~> 2.0.0.rc2"

#config/application.rb
Aws.config = {
    access_key_id: '...',
    secret_access_key: '...',
    region: 'us-west-2'
}

#config/initializers/s3.rb
S3 = Aws::S3.new
S3 = Aws.s3

Then you'll be able to use the API resources to help retrieve objects:

#controller
# yields once per response, even works with non-paged requests
s3.list_objects(bucket:'aws-sdk').each do |resp|
  puts resp.contents.map(&:key)
end

CORS

If you were thinking of xhring into a server, you need to ensure you have the correct CORS permissions to do so

Considering you're wanting to use S3, I would look at this documentation to ensure you set the permissions correctly. This does not apply to the API or an HTTP request (only Ajax)

Upvotes: 1

Related Questions