CODEWITHSUNDEEP
pythonpython-2.7python-requests
Sravani Manukonda
Sravani Manukonda

Reputation: 102

Retry for python requests module hanging

I had to use the request module to fetch a huge number of URLs. Due to network errors, I wanted to implement the Retry mechanism. So my code looks like this:

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
import requests

session = requests.Session()
retry = Retry(total=1,backoff_factor=0.5,status_forcelist=(500,502,504,503))
adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://',adapter)        
head_response = session.get(url,timeout=2)

However, the code keeps hanging when the URL is: tiffins.NET. A normal requests.get with timeout=2 gives 503 status code but doesn't hang.

Am I doing something wrong?

I'm using python 2.7.15rc1.

Upvotes: 1

Views: 1170

Answers (1)

Oleg Polosin
Oleg Polosin

Reputation: 101

Just had the same problem and figured out that it's because of the website's header "Retry-After". By default, Retry uses that value to wait until the next request. To ignore it, set the respect_retry_after_header parameter to False when creating a Retry object.

Upvotes: 7

Related Questions