CODEWITHSUNDEEP
pythonpython-requests
rkj
rkj

Reputation: 711

Retry in Python requests

I have below needs for retry:

My Python script uses REQUESTS module to call API & parse JSON response.

lately the API returns incomplete data like below

Incomplete data:

  {
    offerInfo: {
      siteID: "001",
      language: "en_US",
      currency: "USD"
    },
    userInfo: {
      persona: {
        personaType: "OTHERS"
      },
      userId: "111"
    },
    offers: { }
  }

Complete data:

  {
    offerInfo: {
      siteID: "001",
      language: "en_US",
      currency: "USD"
    },
    userInfo: {
      persona: {
        personaType: "OTHERS"
      },
      userId: "111"
    },
    offers: { 
      Flight: [
        {
          offerDateRange: {
            travelStartDate: [2016, 5, 7],
            travelEndDate: [2016, 5, 11]
          }
       }
    }
  }

A complete response should have ['offers']['Flight'] keys in it ,if not my code should try to call the API N times before it gives up & go to the next API URL.

I am trying with below code with session,but not sure how can I add the check in it ['offers']['Flight']

Note: There is no HTTP error while getting incomplete data returned from API ,so I have to rely on keys ['offers']['Flight'] 

session = requests.Session()
session.mount("http://", requests.adapters.HTTPAdapter(max_retries=2))
session.mount("https://", requests.adapters.HTTPAdapter(max_retries=2))
try:
    response = session.get(url=line,timeout=(connect_timeout,read_timeout),verify=False)

Upvotes: 2

Views: 11293

Answers (2)

samuel161
samuel161

Reputation: 281

You can use tenacity.

Doc: https://tenacity.readthedocs.io/en/latest/

It can log before or after

import logging

logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)

logger = logging.getLogger(__name__)

@retry(stop=stop_after_attempt(3), before=before_log(logger, logging.DEBUG))
def raise_my_exception():
    raise MyException("Fail")

Upvotes: 0

bgreen-litl
bgreen-litl

Reputation: 464

A while loop plus a count as suggested in the comments is the simple answer and may well be good enough.

If instead you'd like to use a module catered to this sort of thing, backoff is a library for decorating functions for retry with configurable backoff behavior. In particular, the on_predicate decorator allows you define criteria for retry based on the return value of a specified function.

import backoff

def has_flight_offer(dct):
    return 'Flight' in dct.get('offers')

@backoff.on_predicate(backoff.constant, has_flight_offer, max_tries=3, interval=1)
def get_offers(session, url))
    response = session.get(url, verify=False)
    return response.json()

It also handles retrying on exceptions and it plays well with requests, so you can use it to handle requests exceptions while you're at it with similar decoration.

@backoff.on_predicate(backoff.constant, has_flight_offer, max_tries=3, interval=1)
@backoff.on_exception(backoff.expo, requests.exceptions.RequestException, max_tries=3)
def get_offers(session, url))
    response = session.get(url, verify=False)
    return response.json()

Upvotes: 2

Related Questions