Using mechanize with watir + phantomjs

Question

I'm trying to insert the html generated from phantom js into a mechanize object so that I can easily search it. I've tried the following to no avail...

b = Watir::Browser.new :phantomjs
url = "www.google.com"
b.goto url

agent = Mechanize.new
#Following is not executed at same time...
#Error 1: lots of errors
page = agent.get(b.html)

#Error 2: `parse': wrong number of arguments (1 for 3) (ArgumentError)
page = agent.parse(b.html)

#Error 3 last ditch effort: undefined method `agent' 
page = agent(b.html)

As I think it through I'm beginning to wonder if I can mechanize an existing html object... I initially got onto it via: http://shane.in/2014/01/headless-web-scraping/ & http://watirmelon.com/2013/02/05/watir-webdriver-with-ghostdriver-on-osx-headless-browser-testing/

Answer 1

I was in the same situation. I write a lot of code with Mechanize so that I do not want to move to nokogiri when using watir. Below code is how I did.

require 'watir'
require 'mechanize'

b = Watir::Browser.new
b.goto(url)
html = b.html
a = Mechanize.new
page = Mechanize::Page.new(nil, {'content-type'=>'text/html'}, html, nil, a)

You could use page to search for elements.

Answer 2

require 'watir'
require 'nokogiri'

b = Watir::Browser.new :phantomjs
url = "http://google.com"
b.goto url

p Nokogiri::HTML(b.html)

You are probably better off just using Nokogiri for this [that is, if you only need to search for some data in source].