OpenSSL.SSL.SysCallError: (10054, 'WSAECONNRESET') during GET request to ArcGIS API

Question

Versions

Python                3.7.0

pyOpenSSL             18.0.0
requests              2.19.1
requests-futures      0.9.7
urllib3               1.23

https://hazards.fema.gov/gis/nfhl/rest/services/public/NFHL/MapServer/identify

I am unable to receive a full response from an ArcGIS rest API when using Python. I am able to navigate to the URL via Chrome and am also able to make the GET request in Postman without issue. The link above is an example of a successful GET request with the JSON response I am expecting. I have tried writing the request using requests, urllib, and socket. Obviously, both requests and socket both fail at recv_into. When running the below code, I receive exactly two chunks before I get the WSAECONNRESET error. If I make a request with geometry set to false, the response is much smaller and returns without issue. I am at a loss for why the connection is only closing in Python and could really use some help fixing it or an alternative to make the query.

Thanks!

Code

def FEMA_identify(self, latitude, longitude, layers, is_envelope=False,
                  envelope_range=None, return_geometry=False):
    envelope = self.getEnvelope(envelope_range, latitude, longitude) if\
        is_envelope else self.getPoint(latitude, longitude)

    extent = envelope if is_envelope else\
        self.getEnvelope(100, latitude, longitude)

    geometry_type =\
        "esriGeometryEnvelope" if is_envelope else "esriGeometryPoint"

    payload = {
        "imageDisplay": "1000,1000",
        "geometrytype": geometry_type,
        "geometry":
            f"{envelope.x_min},{envelope.y_min},{envelope.x_max},{envelope.y_max}",
        "tolerance": f"{0}",
        "mapExtent":
            f"{extent.x_min},{extent.y_min},{extent.x_max},{extent.y_max}",
        "f": "JSON",
        "layers": "visible:{}".format(", ".join(map(str, layers))),
        "returnGeometry": f"{return_geometry}"
    }

    url = 'https://hazards.fema.gov' +\
        '/gis/nfhl/rest/services/public/NFHL/MapServer/identify'

    s = requests.Session()
    r = s.get('https://hazards.fema.gov')
    r = s.get(url, params=payload, stream=True)
    response = b''
    idx = 1
    for chunk in r.iter_content(chunk_size=1024):
        if chunk:
            print(f"Received chunk {idx}")
            response += chunk
            idx += 1
    r.close()

Traceback

Traceback (most recent call last):
  File "C:\VirtualEnvs\tfia\lib\site-packages\urllib3\contrib\pyopenssl.py", line 285, in recv_into
    return self.connection.recv_into(*args, **kwargs)
  File "C:\VirtualEnvs\tfia\lib\site-packages\OpenSSL\SSL.py", line 1814, in recv_into
    self._raise_ssl_error(self._ssl, result)
  File "C:\VirtualEnvs\tfia\lib\site-packages\OpenSSL\SSL.py", line 1631, in _raise_ssl_error
    raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (10054, 'WSAECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\VirtualEnvs\tfia\lib\site-packages\urllib3\response.py", line 331, in _error_catcher
    yield
  File "C:\VirtualEnvs\tfia\lib\site-packages\urllib3\response.py", line 413, in read
    data = self._fp.read(amt)
  File "C:\Program Files\Python37\Lib\http\client.py", line 447, in read
    n = self.readinto(b)
  File "C:\Program Files\Python37\Lib\http\client.py", line 491, in readinto
    n = self.fp.readinto(b)
  File "C:\Program Files\Python37\Lib\socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "C:\VirtualEnvs\tfia\lib\site-packages\urllib3\contrib\pyopenssl.py", line 290, in recv_into
    raise SocketError(str(e))
OSError: (10054, 'WSAECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\VirtualEnvs\tfia\lib\site-packages\requests\models.py", line 749, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "C:\VirtualEnvs\tfia\lib\site-packages\urllib3\response.py", line 465, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "C:\VirtualEnvs\tfia\lib\site-packages\urllib3\response.py", line 430, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "C:\Program Files\Python37\Lib\contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\VirtualEnvs\tfia\lib\site-packages\urllib3\response.py", line 349, in _error_catcher
    raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: OSError("(10054, \'WSAECONNRESET\')")', OSError("(10054, 'WSAECONNRESET')"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\VirtualEnvs\tfia\lib\site-packages\flask\app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\VirtualEnvs\tfia\lib\site-packages\flask\app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "C:\VirtualEnvs\tfia\lib\site-packages\flask\app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "C:\VirtualEnvs\tfia\lib\site-packages\flask\_compat.py", line 35, in reraise
    raise value
  File "C:\VirtualEnvs\tfia\lib\site-packages\flask\app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\VirtualEnvs\tfia\lib\site-packages\flask\app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "C:\Users\rallison\Documents\GitHub\PROCESSING-AUTO-UNDERWRITING-LOGIC\auto_underwriting\receiving.py", line 19, in applications
    processApplication(payload)
  File "C:\VirtualEnvs\tfia\lib\site-packages\celery\local.py", line 191, in __call__
    return self._get_current_object()(*a, **kw)
  File "C:\VirtualEnvs\tfia\lib\site-packages\celery\app\task.py", line 375, in __call__
    return self.run(*args, **kwargs)
  File "C:\Users\rallison\Documents\GitHub\PROCESSING-AUTO-UNDERWRITING-LOGIC\auto_underwriting\processing.py", line 13, in processApplication
    processed = Underwriter(coordinates["lat"], coordinates["lng"])
  File "C:\Users\rallison\Documents\GitHub\PROCESSING-AUTO-UNDERWRITING-LOGIC\auto_underwriting\underwriting.py", line 33, in __init__
    self.underwrite(latitude, longitude)
  File "C:\Users\rallison\Documents\GitHub\PROCESSING-AUTO-UNDERWRITING-LOGIC\auto_underwriting\underwriting.py", line 111, in underwrite
    self.search_range, True)["results"]
  File "C:\Users\rallison\Documents\GitHub\PROCESSING-AUTO-UNDERWRITING-LOGIC\auto_underwriting\underwriting.py", line 96, in FEMA_identify
    r = s.get(url, params=payload, stream=False, timeout=30)
  File "C:\VirtualEnvs\tfia\lib\site-packages\requests\sessions.py", line 525, in get
    return self.request('GET', url, **kwargs)
  File "C:\VirtualEnvs\tfia\lib\site-packages\requests\sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\VirtualEnvs\tfia\lib\site-packages\requests\sessions.py", line 662, in send
    r.content
  File "C:\VirtualEnvs\tfia\lib\site-packages\requests\models.py", line 827, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "C:\VirtualEnvs\tfia\lib\site-packages\requests\models.py", line 752, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: OSError("(10054, \'WSAECONNRESET\')")', OSError("(10054, 'WSAECONNRESET')"))

Answer 1

I ran into a similar issue using an internal server. I fixed it by downgrading openssl from 1.1.1 to 1.0.2.

Alternatively, an easier fix was to not use OpenSSL. Immediately after importing requests call requests.packages.urllib3.contrib.pyopenssl.extract_from_urllib3() and python's stdlib SSL implementation will be used instead of OpenSSL.

It's definitely a bug, but I'm not sure if it's openssl, pyopenssl, or a bad server config.