CODEWITHSUNDEEP
pythonpython-requests
Programmer120
Programmer120

Reputation: 2592

handling exceptions from python requests

My function is executed with threads:

def getdata(self, page, ...):
    tries = 10
    for n in range(tries):
        try:
            ...
            datarALL = []
            url = 'http://website/...'.format(...)
            responsedata = requests.get(url, data=data, headers=self.hed, verify=False)
            responsedata.raise_for_status()
            if responsedata.status_code == 200:  # 200 for successful call
                ...
                    if ...
                        break   
        except (ChunkedEncodingError, requests.exceptions.HTTPError) as e:
            print ("page #{0} run #{1} failed. Returned status code {2}. Reason: {3}. Msg: {4}. Retry.".format(page, n, responsedata.status_code, responsedata.reason, sys.exc_info()[0]))
            if n == tries - 1:
                print ("page {0} could not be imported. Max retried reached.".format(page))
                os._exit(1)  #One thread max retried - close all threads and 
    return datarALL

as follows:

    with ThreadPoolExecutor(max_workers=num_of_workers) as executor:
        futh = [(executor.submit(self.getdata, page,...)) for page in pages]
        for data in as_completed(futh):
            datarALL.extend(data.result())
    print ("Finished generateing data.")
    return datarALL

Sometimes I get unexpected exceptions like: ConnectionResetError: [Errno 104] Connection reset by peer which shuts down my program. I want to change my code so that no matter which exception happens the thread will allways retry untill the if n == tries - 1: is met. I don't want my thread to shut down due to random exception.

I read the request exceptions info page but I don't see how I can catch all the exceptions without manually listing all of them. Is there a generic way to do this?

Basicly I want something like:

        except (ALL EXCEPTIONS from Requests) as e:
            print ("page #{0} run #{1} failed. Returned status code {2}. Reason: {3}. Msg: {4}. Retry.".format(page, n, responsedata.status_code, responsedata.reason, sys.exc_info()[0]))
            if n == tries - 1:
                print ("page {0} could not be imported. Max retried reached.".format(page))
                os._exit(1)  #One thread max retried - close all threads and 
    return datarALL

How can I do that?

EDIT: using 

    except Exception as e:
        print ("page #{0} run #{1} failed. Returned status code {2}. Reason: {3}. Msg: {4}. Retry.".format(page, n, responsedata.status_code, responsedata.reason, sys.exc_info()[0]))
        if n == tries - 1:
            print ("page {0} could not be imported. Max retried reached.".format(page))
            os._exit(1)  #One thread max retried - close all threads and 
return datarALL

Does not catch. it gives me this:

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/response.py", line 331, in _error_catcher
    yield
  File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/response.py", line 640, in read_chunked
    chunk = self._handle_chunk(amt)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/response.py", line 595, in _handle_chunk
    returned_chunk = self._fp._safe_read(self.chunk_left)
  File "/usr/lib/python3.5/http/client.py", line 607, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

The loop does not retry. The run is terminated...

EDIT2:

    except requests.exceptions.RequestException as e:
        print ("page #{0} run #{1} failed. Returned status code {2}. Reason: {3}. Msg: {4}. Retry.".format(page, n, responsedata.status_code, responsedata.reason, sys.exc_info()[0]))
        if n == tries - 1:
            print ("page {0} could not be imported. Max retried reached.".format(page))
            os._exit(1)  #One thread max retried - close all threads and 
return datarALL

Also does not catch the ConnectionResetError: [Errno 104] Connection reset by peer listed above.

Upvotes: 2

Views: 12948

Answers (2)

Abhijith N M
Abhijith N M

Reputation: 1

check your settings.py - whether channels added in installed app.

Upvotes: -3

Tryph
Tryph

Reputation: 6209

Catching all exceptions is usually considered a bad practice since it could hide some problems.

That said, Python exception benefit from inheritance and catching a base exception will catch every exception inherited from this base exception.

See the Python standard exception hierarchy for detail.

You can see that the root exception is BaseException, but this one should never be catched since it will catch Ctrl+C interruptions and generator exits. If you want to catch every exceptions types, you could catch Exception.

You could also want to catch only exception from requests. In that case, according to the doc, it can be done by catching the base exception of the requests module: RequestException

If you want to catch both requests exceptions and ConnectionResetError (which is a Python standard exception), you have to specify both in your except clause:

except (requests.exceptions.RequestException,
        ConnectionResetError) as err:
    # some code

Or if you want to be less specific and catch all possible connection errors you can use ConnectionError instead of ConnectionResetError. (see the exceptions hierarchy)

Finally, you may want to react differently to each exception type. In this case, you can do this:

try:
    # something
except ConnectionError as err:
    # manage connection errors
except requests.exceptions.RequestException as err:
    # manage requests errors

Upvotes: 5

Related Questions