CODEWITHSUNDEEP
pythonhttppython-3.xhttp-headersurllib2
Garfield
Garfield

Reputation: 2537

Get content of http request as well as the response url in single request

How can I get the content of http response as well as the response url(Not the requested url) in single request.

To get response I used:

from urllib2 import Request,urlopen
try:
    headers  = { 'User-Agent' : 'Mozilla/5.0 (X11; U; Linux i686; en-US;)' }
    request  = Request(url, data, headers)
    print urlopen(request).read()
except Exception, e:
        raise Exception(e)

If I want only headers(Headers will have the response url), I used

try:
    headers  = { 'User-Agent' : 'Mozilla/5.0 (X11; U; Linux i686; en-US;)' }
    request  = Request(url, data, headers)
request.get_method = lambda : 'HEAD'
    print urlopen(request).geturl()
    except Exception, e:
        raise Exception(e)

I am making two request to get content & url. How I can get both in a single request. If my function returns content & url as tuple that would be better.

Upvotes: 0

Views: 6123

Answers (2)

aychedee
aychedee

Reputation: 25579

I would refactor your code into something like this. I don't know why you would want to catch the exception only to raise it again without doing anything to it. 

from urllib2 import Request,urlopen

headers  = { 'User-Agent' : 'Mozilla/5.0 (X11; U; Linux i686; en-US;)' }
request  = Request(url, data, headers)
request.get_method = lambda : 'GET'
response = urlopen(request)
return response.read(), response.get_url()

If you do want to catch the exception. You should put it only around the urlopen call.

Upvotes: 0

Dave Lasley
Dave Lasley

Reputation: 6252

If you assign the urlopen(request) to a variable, you can use the both of the attributes with a single request

response = urlopen(request)
request_body = response.read()
request_url  = response.geturl()
print 'URL: %s\nRequest_Body: %s' % ( request_url, request_body )

Upvotes: 4

Related Questions