Webscraping with Python

Question

I'm using this website (http://gasbuddy.com/) to collect gasoline prices. Basically, I want to write a python script that will input zip codes into the search box at the top of the page, and then scrape the results off the next page. I'm stuck at the first step, which is inputting the zip code I want into the form. This is what I have so far:

from mechanize import Browser
import urllib2

br = Browser()
baseURL = "http://www.gasbuddy.com/"
br.open(baseURL)

zipcode = "20010"

forms = [f for f in br.forms()]
print forms[0]
control = forms[0].find_control("ctl00$Content$GBZS$txtZip")
forms[0]["ctl00$Content$GBZS$txtZip"] = "20010"
br.form = forms[0]
page = br.submit()
content = page.read()
br.geturl()

Unfortunately when I submit the form, br.geturl() tells me that I haven't gotten to the page that I want (the url should look something like "http://www.washingtondcgasprices.com/index.aspx?area=Washington%20-%20NE&area=Washington%20-%20NW&area=Washington%20-%20SE&area=Washington%20-%20SW")

If you have any guidance I'd appreciate it. Thanks!

Answer 1

You can do it with Selenium:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

baseURL = "http://www.gasbuddy.com/"

browser = webdriver.Firefox()
zipcode = "20010"

browser.get(baseURL)
elem = browser.find_element_by_id("ctl00_Content_GBZS_txtZip").send_keys(zipcode)
elem = browser.find_element_by_id("ctl00_Content_GBZS_btnSearch").click()

If you want to stick to mechanize, you might want to tweak your browser a bit. But I still suspect it's the javascript that is killing you there. The solution then would be "read the javascript yourself and simulate with mechanize what it would be doing".