Google Cloud Storage api performance regressions

Question

I have a python 3.5 app that makes calls to Google Cloud Storage using the python sdk.

Every once in a while, for 10-30 minutes, all calls to the API fail with BrokenPipeError or ssl.SSLError errors. After some time, they just start working again, I have not noticed a pattern as to why.

Is this a known issue? Is it specific to the python sdk or is this a real performance regression on the side of google?

It should also be noted that these errors will emanate from the same code running on my local machine as well as from a GCE machine.

The trace for BrokenPipe:

Traceback (most recent call last):
  File "oauth2client/util.py", line 140, in positional_wrapper
              return wrapped(*args, **kwargs)
  File "googleapiclient/http.py", line 722, in execute
                                     body=self.body, headers=self.headers)
  File "oauth2client/client.py", line 596, in new_request
                                           redirections, connection_type)
  File "httplib2/__init__.py", line 1314, in request
                      (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "httplib2/__init__.py", line 1064, in _request
          (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "httplib2/__init__.py", line 988, in _conn_request
                  conn.request(method, request_uri, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1083, in request
          self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1128, in _send_request
          self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1079, in endheaders
          self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 911, in _send_output
          self.send(msg)
  File "/usr/lib/python3.5/http/client.py", line 885, in send
              self.sock.sendall(data)
  File "/usr/lib/python3.5/ssl.py", line 886, in sendall
                  v = self.send(data[count:])
  File "/usr/lib/python3.5/ssl.py", line 856, in send
              return self._sslobj.write(data)
  File "/usr/lib/python3.5/ssl.py", line 581, in write
          return self._sslobj.write(data)
BrokenPipeError[Errno 32] Broken pipe

The trace for ssl.SSLError:

  File "oauth2client/util.py", line 140, in positional_wrapper
              return wrapped(*args, **kwargs)
  File "googleapiclient/http.py", line 722, in execute
                                     body=self.body, headers=self.headers)
  File "oauth2client/client.py", line 596, in new_request
                                           redirections, connection_type)
  File "httplib2/__init__.py", line 1314, in request
                      (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "httplib2/__init__.py", line 1064, in _request
          (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "httplib2/__init__.py", line 1017, in _conn_request
                  response = conn.getresponse()
  File "/usr/lib/python3.5/http/client.py", line 1174, in getresponse
                  response.begin()
  File "/usr/lib/python3.5/http/client.py", line 282, in begin
              version, status, reason = self._read_status()
  File "/usr/lib/python3.5/http/client.py", line 243, in _read_status
          line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
                  return self._sock.recv_into(b)
  File "/usr/lib/python3.5/ssl.py", line 924, in recv_into
              return self.read(nbytes, buffer)
  File "/usr/lib/python3.5/ssl.py", line 786, in read
              return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.5/ssl.py", line 570, in read
              v = self._sslobj.read(len, buffer)
ssl.SSLError[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:1974)

Answer 1

Definitely looks like an intermittent issue on Googles side. The broken pipe issue relates to httpclient2 being unable to reconnect an existing connection to their API, this is the error that has the greatest impact on our services. We also on few occasions received "503 Backend Error".

Our "solution" was to basically allow the connections to close themselves by releasing the client once done and creating a new one for the next request.

Bare in mind though that our requests are very sparse, services using Cloud Storage as primary storage probably wants to keep the connections open for as long as possible.