CODEWITHSUNDEEP
pythonweb-scraping
Jason
Jason

Reputation: 1569

Making a post request in python for scraping

My goal is to be able to access the data from a website after inputting information in a field and hitting submit. I'm using Httpfox to grab which values are needed to "post". I included a screenshot of that below the code.

#SECTION 1: import modules
import requests

#SECTION 2: setup variables

url = 'http://www.clarkcountynv.gov/Depts/assessor/Pages/PropertyRecords.aspx?H=redrock&P=assrrealprop/pcl.aspx'

ses = requests.session()

values = []

values.append({
    '__LASTFOCUS' : '',
    '__EVENTARGUMENT' : '',
    '__EVENTTARGET' : 'Submit1',
    '__VIEWSTATE' : '/wEPDwUJLTcyMjI2NjUwZBgBBR5fX0NvbnRyb2xzUmVxdWlyZVBvc3RCYWNrS2V5X18WAgUKY2hrQ3VycmVudAUKY2hrSGlzdG9yeUfXtwoelaE/eJmc1s9mHzvIqqwk',
    '__VIEWSTATEGENERATOR' : '5889EE07',
    '__EVENTVALIDATION' : '/wEWBQLnuv+BDALRqf+zDQK+zcmQBAK+zeWoCALVo8avDjkJwx8mhBoXL3mYGKBSY5lYBPxY',
    'hdnInstance' : 'pcl',
    'parcel' : '124-32-816-087', #this is what changes
    'age' : 'pcl17',
    'Submit1' : 'Submit'})

#SECTION 3: grab html text       
r = ses.post(url, data = values)
r.content() #this either throws an error 'too many values to unpack' or gives me the content of the main page if i play around with the input values a little, not the redirected page which is my problem

enter image description here

Upvotes: 1

Views: 1165

Answers (1)

beppe9000
beppe9000

Reputation: 1114

You don't need to embark in a POST request when you can get the same result with a simple GET

A page inspection of the final page higlights there is an iframe used to show the actual search result.

You can get the result straight from this URL, by replacing "your-parcel-number-here" with the desired value (In your example it was 124-32-816-087).

http://sandgate.co.clark.nv.us/assrrealprop/ParcelDetail.aspx?hdnParcel=your-parcel-number-here

Looks like no cookie is needed and hotlinking works (I tried that link in firefox private mode).

Upvotes: 1

Related Questions