CODEWITHSUNDEEP
pythonpython-2.7urllib2urllib
nKn
nKn

Reputation: 13761

urllib2: Fetch URL content even on Exception

I'm sending a POST request to some URL and this URL then throws a 200 OK or 401 Unauthorized code depending on the parameters provided in the POST request.

Additionally to that return code, the website also returns a text, which is especially useful on errors so the person who did the request knows why it failed. For that, I use this code:

#/usr/bin/env python

import urllib
import urllib2

url = 'https://site/request'
params = {
  'param1': 'value1',
  'param2': 'value2',
  ...
}

data = urllib.urlencode(params)
req = urllib2.Request(url, data)

try:
  response = urllib2.urlopen(req)
  the_page = response.read()
except urllib2.URLError as e:
  print e.code, e.reason  # Returns only 401 Unauthorized, not the text

When the request is successful, I get a 200 code and I can grab the message with the the_page variable. Pretty useless in that case.

But when it fails, the line which throws the URLError is the one calling urlopen(), so I can't grab the web error message.

Is there any way to grab the message even on a URLError event? If not, is there an alternative way to do a POST request and grab the Web content on error?

The Python version is 2.7.6 in my case.

Thanks

Upvotes: 1

Views: 465

Answers (2)

zsoobhan
zsoobhan

Reputation: 1524

I would suggest using the requests library (install using pip install requests)

import requests
url = 'https://site/request'
params = {
  'param1': 'value1',
  'param2': 'value2',
}
post_response = requests.post(url, json=params)


if post_response.ok:
    the_page = post_response.content
    # do your stuff here

print post_response.content  # this will give you the message regardless of failure
print post_response.status_code  # this will give you the status code of the request
post_response.raise_for_status()  # this will throw an error if the status is not 200

Docs: http://docs.python-requests.org/en/latest/

Upvotes: 2

Anton Strogonoff
Anton Strogonoff

Reputation: 34032

In case you catch an HTTPError—it’s a more specific subclass of URLError and I think it would be raised in case of a 401—it can be read as a file-like object, yielding page contents:

Upvotes: 2

Related Questions