CODEWITHSUNDEEP
python-3.xhttpgoogle-cloud-platformgoogle-cloud-functions
Dave
Dave

Reputation: 143

How to prevent Connection reset by peer in a http python request in GCP

The following request is causing a connection reset error. This does not fail every time but sporadically occurs. Any ideas on how to prevent this is appreciated.

http.request("https://dataflow.googleapis.com/v1b3/projects/%s/templates:launch?gcsPath=%s&location=us-central1" % (project, DATAFLOW_SPANNER_EXPORT),
        method="POST",
        headers={'Accept': 'application/json', 'Content-Type': 'application/json'},
        body=body )

This is the error being received. Our function is named create_spanner_export which is launching a dataflow job. This does not fail every time but sporadically occurs. When it fails, the dataflow job is not launched. 

 File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 383, in run_background_function
    _function_handler.invoke_user_function(event_object)
  File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 217, in invoke_user_function
    return call_user_function(request_or_event)
  File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 214, in call_user_function
    event_context.Context(**request_or_event.context))
  File "/user_code/main.py", line 678, in database_export_main
    handle_export(logger, attrib, max_exports, output_dir)
  File "/user_code/main.py", line 526, in handle_export
    spanner_export(logger, attrib, max_exports, output_dir)
  File "/user_code/main.py", line 342, in spanner_export
    for d in database_list[:max_exports]]
  File "/user_code/main.py", line 342, in <listcomp>
    for d in database_list[:max_exports]]
  File "/user_code/main.py", line 274, in create_spanner_export
    body=body )
  File "/env/local/lib/python3.7/site-packages/oauth2client/transport.py", line 175, in new_request
    redirections, connection_type)
  File "/env/local/lib/python3.7/site-packages/oauth2client/transport.py", line 282, in request
    connection_type=connection_type)
  File "/env/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1953, in request
    cachekey,
  File "/env/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1618, in _request
    conn, request_uri, method, body, headers
  File "/env/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1525, in _conn_request
    conn.request(method, request_uri, body, headers)
  File "/opt/python3.7/lib/python3.7/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/python3.7/lib/python3.7/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/opt/python3.7/lib/python3.7/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/python3.7/lib/python3.7/http/client.py", line 1055, in _send_output
    self.send(chunk)
  File "/opt/python3.7/lib/python3.7/http/client.py", line 977, in send
    self.sock.sendall(data)
  File "/opt/python3.7/lib/python3.7/ssl.py", line 1015, in sendall
    v = self.send(byte_view[count:])
  File "/opt/python3.7/lib/python3.7/ssl.py", line 984, in send
    return self._sslobj.write(data)
ConnectionResetError: [Errno 104] Connection reset by peer

Upvotes: 0

Views: 2850

Answers (1)

saccodd
saccodd

Reputation: 1067

Using retries should work: 

@retry(stop=stop_after_attempt(3), wait=wait_random(min=1, max=2))
def sendReq(...):
    http.request("https://dataflow.googleapis.com/v1b3/projects/%s/templates:launch?gcsPath=%s&location=us-central1" % (project, DATAFLOW_SPANNER_EXPORT),
    method="POST",
    headers={'Accept': 'application/json', 'Content-Type': 'application/json'},
            body=body )

See more here: https://cloud.google.com/functions/docs/bestpractices/retries

Upvotes: 2

Related Questions