CODEWITHSUNDEEP
pythonhttprequesturllib
Martin Trigaux
Martin Trigaux

Reputation: 5411

Retrieve HTTPResponse even in case of error

In Python (using Python 3.2 but I guess it should be mostly the same in Python 2.x), I try to do a request to a certain URL.

In the case of error such as Access Denied, I get an exception:

>>> request = urllib.request.urlopen(myurl)
...
  File "/usr/lib/python3.2/urllib/request.py", line 495, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized

But I would like to see the headers of the request even in case of error.

>>> request = urllib.request.urlopen(myurl)
>>> print(request.status)
401
>>> print(request.headers)
...

I also noticed that when a page replies with a redirection status code such as 301, the response I get is the redirected page, not the first one (which is the one I want).

Any idea how I can do that?

Upvotes: 2

Views: 517

Answers (1)

Maria Zverina
Maria Zverina

Reputation: 11173

Have you considered using the requests package? It gives you history of all the redirects that are made to fulfil your request:

>>> import requests
>>> r = requests.get('http://google.com')
>>> r
<Response [200]>
>>> r.history
[<Response [301]>, <Response [302]>]
>>> r.url
u'http://www.google.co.uk/'

And it also deals sanely with 401 errors

>>> r = requests.get('http://sitesurgeon.co.uk/!dev/http-authorisation/staff/index.htm')
>>> r
<Response [401]>
>>> r.content
'<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"> ...
....'
>>> r.headers
{'date': 'Wed, 06 Jun 2012 14:24:16 GMT', 'x-powered-by': 'PHP/5.3.13', 'transfer-encoding': 'chunked', 'content-type': 'text/html; charset=utf-8', 'www-authenticate': 'Basic realm="Staff Area"', 'server': 'Apache'}

If you wish to control your timeout simply make the request as follows:

requests.get('http://google.com', timeout=0.1)

Upvotes: 4

Related Questions