CODEWITHSUNDEEP
pythonurllibhttp-status-code-429
Bijan
Bijan

Reputation: 8586

Python HTTP Error 429 with urllib2

I am using the following code to resolve redirects to return a links final url

def resolve_redirects(url):
    return urllib2.urlopen(url).geturl()

Unfortunately I sometimes get HTTPError: HTTP Error 429: Too Many Requests. What is a good way to combat this? Is the following good or is there a better way.

def resolve_redirects(url):
    try:
        return urllib2.urlopen(url).geturl()
    except HTTPError:
        time.sleep(5)
        return urllib2.urlopen(url).geturl()

Also, what would happen if there is an exception in the except block?

Upvotes: 2

Views: 11192

Answers (3)

Joe
Joe

Reputation: 392

Adding User-Agent to request header solved my issue:

from urllib import request
from urllib.request import urlopen

url = 'https://www.example.com/abc.json'
req = request.Request(url)
req.add_header('User-Agent', 'abc-bot')
response = request.urlopen(req)

Upvotes: 0

Jesse W at Z - Given up on SE
Jesse W at Z - Given up on SE

Reputation: 1748

It would be better to make sure the HTTP code is actually 429 before re-trying.

That can be done like this:

def resolve_redirects(url):
    try:
        return urllib2.urlopen(url).geturl()
    except HTTPError, e:
        if e.code == 429:
             time.sleep(5);
             return resolve_redirects(url)
        raise

This will also allow arbitrary numbers of retries (which may or may not be desired).

https://docs.python.org/2/howto/urllib2.html#httperror

Upvotes: 4

Gnarlywhale
Gnarlywhale

Reputation: 4220

This is a fine way to handle the exception, though you should check to make sure you are always sleeping for the appropriate amount of time between requests for the given website (for example twitter limits the amount of requests per minute and has this amount clearly shown in their api documentation). So just make sure you're always sleeping long enough.

To recover from an exception within an exception, you can simply embed another try/catch block:

def resolve_redirects(url):
    try:
        return urllib2.urlopen(url).geturl()
    except HTTPError:
        time.sleep(5)
        try:
            return urllib2.urlopen(url).geturl()
        except HTTPError:
            return "Failed twice :S"

Edit: as @jesse-w-at-z points out, you should be returning an URL in the second error case, the code I posted is just a reference example of how to write a nested try/catch.

Upvotes: 1

Related Questions