Scraping data from a site that has no form tag but a text input using python

Question

I'm working on a python program to scrape data from here. I've had successes before but this one poses a challenge for me. I am using beautiful soup and mechanize. I need to be able to input a zip code in the text box to produce the results here's I'm after.

Here's the snippet that contains the input text box:


		
        
            Address, City or Zip:
            
                  within  
            
                miles
            
        
        
               
            
        
        
        
        
            Only show AASM SleepTM capable sleep centers in my state
            What is AASM SleepTM?

So far these are my attempts

url = 'http://www.sleepeducation.org/find-a-facility'
MILES = '100'
CODE = '33060'

attempt one

first = urllib2.Request(url,
                   data=urllib.urlencode({'value': CODE}),
                   headers={'User-Agent' : 'Google Chrome'                             'Cookie': 'name = ctl00$ContentPlaceHolder1$C001$tbUserAddress'})

attempt two

post_params = {
       'ctl00$ContentPlaceHolder1$C001$tbUserAddress': CODE
}
first = urllib.urlencode(post_params)

driver = webdriver.Chrome()
driver.get(url)
sbox = driver.find_element_by_class_name("ctl00$ContentPlaceHolder1$C001$tbUserAddress")
sbox.send_keys(CODE)
        driver.find_element_by_class_name("ctl00$ContentPlaceHolder1$C001$btnSearchClient").click()

attempt 3

br = mechanize.Browser()
br.open(url)
br.select_form(name='ctl00$ContentPlaceHolder1$C001$tbUserAddress')
br['value'] = CODE
br.submit()

http = urllib2.urlopen(br.response())
soup = BeautifulSoup(http, "html5lib")

Error = "no form matching name 'ctl00$ContentPlaceHolder1$C001$tbUserAddress'"

attempt 4

soup.find('input', {'name': 'ctl00$ContentPlaceHolder1$C001$tbUserAddress'})['value'] = CODE
soup.find('input', {'name': 'ctl00$ContentPlaceHolder1$C001$btnSearchClient'}).click()

Bohdan Biloshytskiy · Accepted Answer

If i correctly understand your question, you want to send request, with specific params, and check response. Okay, lets look what request where sent after submit. Lets open Postman.Post request params

As we can see ctl00$ContentPlaceHolder1$C001$tbUserAddress get value 100, and ctl00$ContentPlaceHolder1$T6B6681F0008$ddlRadius, ctl00$ContentPlaceHolder1$C001$ddlRadius, ctl00$cphTopBar$T917BC451013$rblRadius got radius value 25.

So let's get a little snippet with data to send post request and get required response

I use python requests

And lxml to parse html response

I prefere lxml, it's a harder to understand but much faster than BeautifulSoup.

import requests
from lxml import html

input_data = {
    'ctl00$cphTopBar$T917BC451013$rblRadius': 25,
    'ctl00$ContentPlaceHolder1$T6B6681F0008$ddlRadius': 25,
    'ctl00$ContentPlaceHolder1$C001$ddlRadius': 25,
    'ctl00$ContentPlaceHolder1$C001$tbUserAddress': 100
}
resp = requests.post('http://www.sleepeducation.org/find-a-facility', data=input_data)
tree = html.fromstring(resp.text)
print(tree.xpath('//div[@id="ContentPlaceHolder1_C001_map_canvas"]')[0])

I dont have enough reputation to put links to documentation, i'll try to put them in comments, or you can just google python requests and python lxml Also U can do it with BeautifulSoup

import BeautifulSoup
import requests

input_data = {
        'ctl00$cphTopBar$T917BC451013$rblRadius': 25,
        'ctl00$ContentPlaceHolder1$T6B6681F0008$ddlRadius': 25,
        'ctl00$ContentPlaceHolder1$C001$ddlRadius': 25,
        'ctl00$ContentPlaceHolder1$C001$tbUserAddress': 100
    }
resp = requests.post('http://www.sleepeducation.org/find-a-facility', data=input_data)
soup = BeautifulSoup.BeautifulSoup(resp.text)
soup.find("div", {"id": "ContentPlaceHolder1_C001_map_canvas"})

Scraping data from a site that has no form tag but a text input using python

Answers (2)

Related Questions