Error using Google Cloud KMS with Flask, gunicorn and gevent worker: "google.api_core.exceptions.ServiceUnavailable: 503 Deadline Exceeded"

Question

Attempting encryption or decryption with google-cloud-kms in the context of a Flask request results in the error below when serving the Flask app with gunicorn and gevent workers. The error happens both when running the app locally and when in the Google App Engine (flexible) environment.

app_1  | Traceback (most recent call last):
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
app_1  |     return callable_(*args, **kwargs)
app_1  |   File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
app_1  |     return _end_unary_response_blocking(state, call, False, None)
app_1  |   File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
app_1  |     raise _Rendezvous(state, None, None, deadline)
app_1  | grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
app_1  |    status = StatusCode.UNAVAILABLE
app_1  |    details = "Deadline Exceeded"
app_1  |    debug_error_string = "{"created":"@1561236057.820157100","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":14}"
app_1  | >
app_1  |
app_1  | The above exception was the direct cause of the following exception:
app_1  |
app_1  | Traceback (most recent call last):
app_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2311, in wsgi_app
app_1  |     response = self.full_dispatch_request()
app_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1834, in full_dispatch_request
app_1  |     rv = self.handle_user_exception(e)
app_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1737, in handle_user_exception
app_1  |     reraise(exc_type, exc_value, tb)
app_1  |   File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 36, in reraise
app_1  |     raise value
app_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1832, in full_dispatch_request
app_1  |     rv = self.dispatch_request()
app_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1818, in dispatch_request
app_1  |     return self.view_functions[rule.endpoint](**req.view_args)
app_1  |   File "/app/app.py", line 14, in encrypt
app_1  |     encrypted = client.encrypt(key_name, plaintext.encode())
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/cloud/kms_v1/gapic/key_management_service_client.py", line 1286, in encrypt
app_1  |     request, retry=retry, timeout=timeout, metadata=metadata
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
app_1  |     return wrapped_func(*args, **kwargs)
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 273, in retry_wrapped_func
app_1  |     on_error=on_error,
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 182, in retry_target
app_1  |     return target()
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
app_1  |     return func(*args, **kwargs)
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
app_1  |     six.raise_from(exceptions.from_grpc_error(exc), exc)
app_1  |   File "<string>", line 3, in raise_from
app_1  | google.api_core.exceptions.ServiceUnavailable: 503 Deadline Exceeded

requirements.txt

Flask==1.0.3
gevent==1.4.0
google-cloud-kms==1.0.0
grpcio==1.21.1
gunicorn==19.9.0

Example Flask app that reproduces this:

from flask import Flask
from google.cloud import kms_v1
import base64


def create_app():
    app = Flask(__name__)

    @app.route('/')
    def encrypt():
        plaintext = "plain"
        client = kms_v1.KeyManagementServiceClient()
        key_name = client.crypto_key_path('project', 'location', 'keyring', 'key')
        encrypted = client.encrypt(key_name, plaintext.encode())
        return base64.urlsafe_b64encode(encrypted.ciphertext)

    return app


example_app = create_app()

Served with:

gunicorn -t 300 -b :8080 -k gevent -w 4 app:example_app

Answer 1

Answering my own question as I spent some time debugging this, and found answers online for similar but not this particular issue.

The issue seems to be how grpc and gevent work (or don't work) together, and the quickest fix was to use another gunicorn worker. Tested that at least sync, eventlet and gthread workers prevent this error from occurring.

gunicorn -t 300 -b :8080 -k eventlet -w 4 app:example_app