CODEWITHSUNDEEP
pythonpython-requests
Walid Saad
Walid Saad

Reputation: 1001

Requests with retries

I'm trying to make requests retry if the response status code is not 200, so I wrote this function:

    def _get_resource(url, max_retries=0, delay=0):
         number_retries = 0
         response = requests.get(url)
         while response.status_code != 200 and number_retries < max_retries:
            time.sleep(delay)
            response = requests.get(url)
            number_retries += 1
        response.raise_for_status()
        return response.content

Any suggestions to improve it?

Upvotes: 3

Views: 3821

Answers (1)

TomDotTom
TomDotTom

Reputation: 6744

You could try using requests HTTPAdapter. Something like the following should work:

import time
import httplib

import requests
from requests.adapters import HTTPAdapter


class RetryHTTPAdapter(HTTPAdapter):

    SECONDS_BETWEEN_RETRIES = 5

    def __init__(self, retry_time=120, *args, **kwargs):
        self.retry_time = retry_time
        super(RetryHTTPAdapter, self).__init__(*args, **kwargs)

    def send(self, *args, **kwargs):
        for _ in range(int(self.retry_time / self.SECONDS_BETWEEN_RETRIES)):
            response = super(RetryHTTPAdapter, self).send(*args, **kwargs)
            if response.status_code == httplib.OK:
                break
            time.sleep(self.SECONDS_BETWEEN_RETRIES)
        return response

s = requests.Session()
s.mount('http://', RetryHTTPAdapter(retry_time=180))
s.mount('https://', RetryHTTPAdapter(retry_time=180))

s.get('http://example.com')

Upvotes: 2

Related Questions