Web scraping HTML in a Rails app

Question

I'm trying to get my rails app to fetch the HTML source of a web page.

I want to get all of the HTML from a URI like /news_articles/7 into a string.

I tried using something like Nokogiri but it seems to lock mutex.

The purpose for this is to send a string of HTML to Amazon's SES.

Thanks

Answer 1

Nokogiri in combination with Mechanize will serve you well.

Gemfile

gem 'nokogiri'
gem 'mechanize'

controller

agent = Mechanize.new()
# allow the agent to follow redirects
agent.follow_meta_refresh = true
# get the desired page
page = agent.get('http://www.mysite.com/new_articles/7')
# output its html
page.body

Possible Duplicate