CODEWITHSUNDEEP
pythonweb-scrapingmechanize
Zack Yoshyaro
Zack Yoshyaro

Reputation: 2124

How do you view the request headers that mechanize is using?

I am attempting to submit some data to a form programatically. I'm having a small issue whereby the server is "not liking" what I'm sending it. Frustratingly, there is no error messages, or anything that could help diagnose the issue, all it does is spit me back to the same page I started on when I hit br.submit().

When I click the submit button manually in the browser, the resulting page shows a small "success!" message. No such message appears when submitting via the script. Additionally, no changes are actually being posted to the server. It's quite strange, and the first time I've encountered this behavior.

Digging through the Mechanize docs, it suggests that under these strange, hard to diagnose issues, that it's best to copy the request headers that are actually submitted by the browser.

My question is, how do I see what the request headers are when I call br.submit()?

Sample Code

location = 'http://ww.mysite.com'

br = mechanize.Browser()
cj = mechanize.LWPCookieJar()
br.set_cookiejar(cj)

username = MY_USER_NAME
password = MY_PASSWORD
br.addheaders.append(('Authorization', 'Basic %s' % base64.encodestring('%s:%s' % (username, password))))

br.open(location)

br.select_form(nr=0)
br['text'] = 'MY JUNK TO SUBMIT'    #Text field. Can put anything
br['DropDown1'] = ['4']             #This is a dropdown of integer values
br['DropDown2'] = ['3']             #Also a dropdown of ints
br.submit()

How do I see which headers are being sent when I submit the form?

Upvotes: 4

Views: 11067

Answers (1)

Christopher Hackett
Christopher Hackett

Reputation: 6192

Are you asking how to see what headers your browser or mechanize is sending?

Browser

Like the other commentators say you can check the headers sent by the browsers with a plugin like Firebug (Firefox), Developer tools (IE 'F12', Chrome Developer tools and Opera Dragonfly) etc.

Mechanize

With mechanize you can get a copy of the headers sent by doing something like 

import mechanize 

br = mechanize.Browser()
br.open("http://stackoverflow.com")
request = br.request
request.header_items()

Which gives in this case 

[('Host', 'stackoverflow.com'), ('User-agent', 'Python-urllib/2.7')]

Other/One off

As always for a one off debug or if nothing is provided then you can use Wireshark to check what headers are been sent over the network. Tip: use a filter like (http.request.uri == "http://stackoverflow.com/")

Upvotes: 10

Related Questions