Geopy Geocoding Requests Not Catching Errors Correctly

Question

I'm using the geopy library to perform geocoding requests with the Nominatim service. Despite setting up detailed error handling, I still receive general errors without catching specific ones like ConnectTimeout, MaxRetryError, and GeocoderUnavailable. I want to ensure that all relevant exceptions are caught and properly logged.

Here is a minimal working example (MWE) of my code.

import logging
import traceback
from geopy.geocoders import Nominatim
from geopy.extra.rate_limiter import RateLimiter
from geopy.exc import GeocoderTimedOut, GeocoderRateLimited, GeocoderUnavailable
from requests.exceptions import ConnectTimeout
from urllib3.exceptions import MaxRetryError, ConnectTimeoutError

# Set up logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class MyGeocoder:
    def __init__(self, user_agent: str):
        self.geolocator = Nominatim(user_agent=user_agent, timeout=10)
        self.geocode = RateLimiter(self.geolocator.geocode, min_delay_seconds=1, max_retries=3, error_wait_seconds=3)

    def get_location(self, address: str):
        try:
            location = self.geocode(address)
            if not location:
                logger.warning(f"No address found for {address}")
                return None
            return location
        except GeocoderRateLimited as e:
            logger.warning(f"Geocoder rate limited for {address}: {str(e)}
{traceback.format_exc()}")
        except (ConnectTimeout, ConnectTimeoutError, MaxRetryError, GeocoderUnavailable, GeocoderTimedOut) as e:
            logger.warning(f"Geocode failed on input {address} with message: {str(e)}
{traceback.format_exc()}")
        except Exception as e:
            logger.error(f"Error handling request for address {address}: {str(e)}
{traceback.format_exc()}")
        return None

# Example usage
if __name__ == "__main__":
    geocoder = MyGeocoder(user_agent="my_geocoder")
    location = geocoder.get_location("Milford Sound, New Zealand")
    if location:
        print(location)
    else:
        print("Failed to get location")

My Issue

Despite the exception handling in place, I still encounter the following error traceback, which seems to bypass my specific error handling blocks:

WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(, 'Connection to nominatim.openstreetmap.org timed out. (connect timeout=10)')': /search?q=Milford+Sound%2C+New+Zealand&format=json&limit=1
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(, 'Connection to nominatim.openstreetmap.org timed out. (connect timeout=10)')': /search?q=Milford+Sound%2C+New+Zealand&format=json&limit=1
WARNING:geopy:RateLimiter caught an error, retrying (0/3 tries). Called with (*('Milford Sound, New Zealand',), **{}).
Traceback (most recent call last):
  File "[...]\Python\Python312\Lib\site-packages\urllib3\connection.py", line 203, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]\Python\Python312\Lib\site-packages\urllib3\util\connection.py", line 85, in create_connection
    raise err
  File "[...]\Python\Python312\Lib\site-packages\urllib3\util\connection.py", line 73, in create_connection
    sock.connect(sa)
TimeoutError: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "[...]\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 790, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "[...]\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 491, in _make_request
    raise new_e
  File "[...]\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "[...]\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 1096, in _validate_conn
    conn.connect()
  File "[...]\Python\Python312\Lib\site-packages\urllib3\connection.py", line 611, in connect
    self.sock = sock = self._new_conn()
                       ^^^^^^^^^^^^^^^^
  File "[...]\Python\Python312\Lib\site-packages\urllib3\connection.py", line 212, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (, 'Connection to nominatim.openstreetmap.org timed out. (connect timeout=10)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "[...]\Python\Python312\Lib\site-packages
equests\adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "[...]\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 874, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "[...]\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 874, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "[...]\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 844, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "[...]\Python\Python312\Lib\site-packages\urllib3\util
etry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Max retries exceeded with url: /search?q=Milford+Sound%2C+New+Zealand&format=json&limit=1 (Caused by ConnectTimeoutError(, 'Connection to nominatim.openstreetmap.org timed out. (connect timeout=10)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "[...]\Python\Python312\Lib\site-packages\geopy\adapters.py", line 482, in _request
    resp = self.session.get(url, timeout=timeout, headers=headers)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]\Python\Python312\Lib\site-packages
equests\sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]\Python\Python312\Lib\site-packages
equests\sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]\Python\Python312\Lib\site-packages
equests\sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]\Python\Python312\Lib\site-packages
equests\adapters.py", line 507, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Max retries exceeded with url: /search?q=Milford+Sound%2C+New+Zealand&format=json&limit=1 (Caused by ConnectTimeoutError(, 'Connection to nominatim.openstreetmap.org timed out. (connect timeout=10)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "[...]\Python\Python312\Lib\site-packages\geopy\extra
ate_limiter.py", line 136, in _retries_gen
    yield i  # Run the function.
    ^^^^^^^
  File "[...]\Python\Python312\Lib\site-packages\geopy\extra
ate_limiter.py", line 274, in __call__
    res = self.func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]\Python\Python312\Lib\site-packages\geopy\geocoders
ominatim.py", line 297, in geocode
    return self._call_geocoder(url, callback, timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]\Python\Python312\Lib\site-packages\geopy\geocoders\base.py", line 368, in _call_geocoder
    result = self.adapter.get_json(url, timeout=timeout, headers=req_headers)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]\Python\Python312\Lib\site-packages\geopy\adapters.py", line 472, in get_json
    resp = self._request(url, timeout=timeout, headers=headers)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[...]\Python\Python312\Lib\site-packages\geopy\adapters.py", line 494, in _request
    raise GeocoderUnavailable(message)
geopy.exc.GeocoderUnavailable: HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Max retries exceeded with url: /search?q=Milford+Sound%2C+New+Zealand&format=json&limit=1 (Caused by ConnectTimeoutError(, 'Connection to nominatim.openstreetmap.org timed out. (connect timeout=10)'))
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(, 'Connection to nominatim.openstreetmap.org timed out. (connect timeout=10)')': /search?q=Milford+Sound%2C+New+Zealand&format=json&limit=1
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(, 'Connection to nominatim.openstreetmap.org timed out. (connect timeout=10)')': /search?q=Milford+Sound%2C+New+Zealand&format=json&limit=1

Note

The provided MWE will probably work fine for most people because the error typically arises when making many (like 50-100) requests one after another. I understand that there are request limits, so I just want to catch the error accordingly.

Question

Why are the specific errors like ConnectTimeout, MaxRetryError, and GeocoderUnavailable not being caught correctly in my except blocks? How can I ensure that all relevant exceptions are caught and logged with detailed error messages?

I've added the exception handling similar to the question here to try and catch the exceptions, but they do not get catched.

Geopy Geocoding Requests Not Catching Errors Correctly

My Issue

Note

Question

Answers (0)

Related Questions