Handeling errors in Python requests

Question

I am learning to use requests in Python and I need a way to get a meaningful output if the site does not exist at all.

I looked at this question, but it is unclear if the OP of the question actually wants to check if the site exists, or if it just returns an error. The problem with all of the answers that question is that if the site does not exist at all we cannot really use HTTP response headers, because no response is returned from a server that does not exist.

Here is an example.

If I use this code I will not get any errors because the site exists.

import requests
r = requests.get('https://duckduckgo.com')

However, if I enter a web page I know does not exist I will get an error

import requests
r = requests.get('https://thissitedoesnotexist.com')

if r.status_code == requests.codes.ok:
    print('Site good')
else:
    print('Site bad')

This error is super long and I would prefer to have a more meaningful and short error if the site does not exist.

Traceback (most recent call last):
  File "C:\Users\ADMIN\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\ADMIN\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "C:\Users\ADMIN\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "C:\Users\ADMIN\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connection.py", line 416, in connect
    self.sock = ssl_wrap_socket(
  File "C:\Users\ADMIN\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "C:\Users\ADMIN\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "C:\Users\ADMIN\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 512, in wrap_socket
    return self.sslsocket_class._create(
  File "C:\Users\ADMIN\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1070, in _create
    self.do_handshake()
  File "C:\Users\ADMIN\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1341, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:997)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\ADMIN\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\adapters.py", line 439, in send
    resp = conn.urlopen(
  File "C:\Users\ADMIN\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "C:\Users\ADMIN\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='234876.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:997)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\ADMIN\Desktop\tetst.py", line 2, in <module>
    r = requests.get('https://234876.com')
  File "C:\Users\ADMIN\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\ADMIN\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\ADMIN\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\ADMIN\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\ADMIN\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='234876.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:997)')))

Is it possible to make a function that returns, for example print('The site probably does not exist') or at least does not give an EOF error?

Answer 1

Normally the desirable thing to do is trap Exceptions from requests

You also can use .raise_for_status() on the Response to get a meaningful Exception for non-OK requests

However, you want to watch out for where you want to handle an Exception

• immediately? can your program handle it meaningfully or should it exit?
• should the caller handle a specific Exception (such as requests.exceptions.Timeout) or a more general one?
• do you have many functions which call each other? should any handle some subset of possible Exceptions? and which?

See Python Exception Hierarchy for how the first-party Exceptions inheritance structure

import sys
import requests

TIMEOUT_REQUESTS = (5, 30)

def some_function_which_makes_requests(timeout=TIMEOUT_REQUESTS):
    r = requests.get("https://example.com", timeout=timeout)
    r.raise_for_status()  # raise for non-OK
    return r.json()       # interpret response via some method (for example as JSON)

def main():
    ...
    try:
        result_json = some_function_which_makes_requests()
    except requests.exceptions.Timeout as ex:
        #print("WARNING: request timed out")
        #result_json = None  # still effectively handled for later program?
        raise Exception("request timed out") from ex
    except requests.exceptions.RequestException as ex:
        sys.exit(f"something wrong with Request: {repr(ex)}")
    except Exception:
        sys.exit(f"something wrong around Request: {repr(ex)}")
    # now you can use result_json

Answer 2

Did some more research and just learned that I need to use a Python Try Except as mentioned by @Anand Sowmithiran. Here is a video explaining it for beginners: https://www.youtube.com/watch?v=NIWwJbo-9_8

import requests

try:
    r = requests.get("http://www.duckduckgo.com")
except requests.exceptions.ConnectionError:
    print('\n\tSorry. There was a network problem getting the URL. Perhaps it does not exist?\n\tCheck the URL, DNS issues or if you are being rejected by the server.')
else:
    print(r)