CODEWITHSUNDEEP
pythonpython-requests
Giles Knap
Giles Knap

Reputation: 535

Python requests: how to determine response code on MaxRetries exception

I would like to know the cause of the issue when my get or post have error responses until maximum retries are exceeded.

I have an example test below. This throws a MaxRetriesError as expected. However, the original HTTPError seems to be lost.

from requests import Session, exceptions
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
from unittest import TestCase
import logging


class TestRequests(TestCase):

    def test_retries(self):
        session = Session()
        retries = Retry(total=5,
                        backoff_factor=0.1,
                        status_forcelist=[500, 502, 503, 504],
                        method_whitelist=frozenset(['GET', 'POST']))

        session.mount('https://', HTTPAdapter(max_retries=retries))
        session.mount('http://', HTTPAdapter(max_retries=retries))

        try:
            result = session.get('https://httpbin.org/status/500',
                                 stream=True,
                                 timeout=2)
            print(result)
        except exceptions.HTTPError as e:
            logging.error('http', exc_info=e)
        except exceptions.RetryError as e:
            logging.error('retry', exc_info=e)

The Traceback shows the following information, it only goes back as far as the last RetryError and not to the HTTPError. The exception's string does mention the response code but parsing error message strings is not a robust approach.

ERROR:root:retry
Traceback (most recent call last):
  File "/home/giles/venv/gphotos-sync3/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/giles/venv/gphotos-sync3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 731, in urlopen
    body_pos=body_pos, **response_kw)
  File "/home/giles/venv/gphotos-sync3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 731, in urlopen
    body_pos=body_pos, **response_kw)
  File "/home/giles/venv/gphotos-sync3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 731, in urlopen
    body_pos=body_pos, **response_kw)
  [Previous line repeated 2 more times]
  File "/home/giles/venv/gphotos-sync3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 711, in urlopen
    retries = retries.increment(method, url, response=response, _pool=self)
  File "/home/giles/venv/gphotos-sync3/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='httpbin.org', port=443): Max retries exceeded with url: /status/500 (Caused by ResponseError('too many 500 error responses',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/giles/github/gphotos-sync/test/test_requests.py", line 36, in test_retries
    timeout=2)
  File "/home/giles/venv/gphotos-sync3/lib/python3.6/site-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/home/giles/venv/gphotos-sync3/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/giles/venv/gphotos-sync3/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/home/giles/venv/gphotos-sync3/lib/python3.6/site-packages/requests/adapters.py", line 507, in send
    raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool(host='httpbin.org', port=443): Max retries exceeded with url: /status/500 (Caused by ResponseError('too many 500 error responses',))


Ran 1 test in 4.092s

OK

Upvotes: 5

Views: 4193

Answers (1)

Giles Knap
Giles Knap

Reputation: 535

I have found that I can see the original error if I tell Retries not to raise its own error (raise_on_status=False), but then inspect the last response instead. See modified code below.

from requests import Session, exceptions
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
from urllib3.exceptions import MaxRetryError
from unittest import TestCase
import logging


class TestRequests(TestCase):

    def test_retries(self):
        session = Session()
        retries = Retry(total=5,
                        backoff_factor=0.1,
                        status_forcelist=[500, 502, 503, 504],
                        method_whitelist=frozenset(['GET', 'POST']),
                        raise_on_status=False)

        session.mount('https://', HTTPAdapter(max_retries=retries))
        session.mount('http://', HTTPAdapter(max_retries=retries))

        try:
            result = session.get('https://httpbin.org/status/500',
                                 stream=True,
                                 timeout=2)
            print(result)
            result.raise_for_status()

        except exceptions.HTTPError as e:
            logging.error('http', exc_info=e)
        except MaxRetryError as e:
            logging.error('retry', exc_info=e)

This results in:

------------------------------------------------------------- Captured stdout call --------------------------------------------------------------
<Response [500]>
--------------------------------------------------------------- Captured log call ---------------------------------------------------------------
test-requests.py            30 ERROR    http
Traceback (most recent call last):
  File "/home/hgv27681/github/tests/test-requests.py", line 27, in test_retries
    result.raise_for_status()
  File "/home/hgv27681/github/venv3/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url: https://httpbin.org/status/500

Upvotes: 5

Related Questions