CODEWITHSUNDEEP
python
TiOLUWA
TiOLUWA

Reputation: 225

urllib2.urlopen() returning a different result

I'm trying to fill out a form using a python program, it works well for some sites, but not this particular one, i'm not sure why.

this is the code snipet

query = {
'adults':'1',
'children':'0',
'infants':'0',
'trip':'RT',
'deptCode':'LOS',
'arrvCode':'ABV',
'searchType':'D',
'deptYear':'2011',
'deptMonth':'12',
'deptDay':'10',
'retYear':'2011',
'retMonth':'12',
'retDay':'11',
'cabin':'E',
'currency':'NGN',
'deptTime':'',
'arrvTime':'',
'airlinePref':''}

encoded = urllib.urlencode(query)



url = 'http://www.wakanow.com/ng/flights/SearchProcess.aspx?' + encoded
user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
headers = { 'User-Agent' : user_agent }
req = urllib2.Request(url, encoded, headers)

response = urllib2.urlopen(req)

print 'RESPONSE:', response
print 'URL     :', response.geturl()

headers = response.info()
print 'DATE    :', headers['date']
print 'HEADERS :'
print '---------'
print headers

data = response.read()
print 'LENGTH  :', len(data)
print 'DATA    :'
print '---------'
print data

It all works fine, but the result i get is not what appears if i type the entire url into a web browser directly, that gives me the right result.

I'm not sure what the problem is can anyone help me out?

Upvotes: 1

Views: 2190

Answers (3)

John Doe
John Doe

Reputation: 3506

You're likely doing a GET in your browser, but in your code, you're actually doing a POST to a URL with the query data AND with your query data as the POST data. You probably just want to do a GET. From this page,

urllib2.Request(url[, data][, headers][, origin_req_host][, unverifiable])

data may be a string specifying additional data to send to the server, or None if no such data is needed. Currently HTTP requests are the only ones that use data; the HTTP request will be a POST instead of a GET when the data parameter is provided. data should be a buffer in the standard application/x-www-form-urlencoded format. The urllib.urlencode() function takes a mapping or sequence of 2-tuples and returns a string in this format.

So, what you really want, is:

req = urllib2.Request(url, headers=headers)

Upvotes: 2

soField
soField

Reputation: 2696

This url hangs. Try it with a less heavy search string.

And you may consider controlling this with a timeout:

import urllib,urllib2,socket

timeout = 10
socket.setdefaulttimeout(timeout)

Upvotes: 0

jfs
jfs

Reputation: 414745

If the 2nd parameter (data) for urllib2.Request is provided then urllib2.urlopen(req) makes a POST request.

Use encoded either in url (GET) or as data in urllib2.Request (POST) not both i.e,

either GET request:

url = 'http://www.wakanow.com/ng/flights/SearchProcess.aspx?' + encoded
req = urllib2.Request(url, headers=headers) #NOTE: no `encoded`

or POST request:

url = 'http://www.wakanow.com/ng/flights/SearchProcess.aspx' #NOTE: no `encoded`
req = urllib2.Request(url, data=encoded, headers=headers)

Upvotes: 1

Related Questions