CODEWITHSUNDEEP
pythongrpcgrpc-python
paul
paul

Reputation: 1

How to detect gRPC server is down from gRPC AIO python client

I've been facing this issue where I have a gRPC AIO python client sending bunch of configuration changes to the gRPC server, though it's a bi-directional RPC, client is not expecting any message from gRPC server. So whenever there is a configuration change the client sends gRPC message containing the configuration. It keeps the channel open and it's not closed (not calling done_writing()) . But when it doesn't have anything to send, it polls a queue for any new message in a tight loop. During that time if the server goes down, the client is not able to detect that. But, as soon as some data is available in the queue, while pushing it to server, client is able to detect that server went down(exception thrown).

How to detect server went down while there is no data to send and client is waiting for data availability. Is there any gRPC API I can call on the channel while the client is waiting for data to send, in order to detect channel failure(basically if there is any api call which throws exception while the server went down is also good for me, I was not able to find any useful API)? I tried gRPC keepalive, but it didn't work for my scenario.



            async with grpc.aio.insecure_channel('127.0.0.1:51234') as channel:
            stream = hello_pb2.HelloStub(channel).Configure()
            await stream.wait_for_connection()
            while True:
                if queue.empty():
                    continue
                if not queue.empty():
                    item = queue.get()
                    await asyncio.sleep(0.001)
                    queue.task_done()
                    await stream.write(item)
                await asyncio.sleep(0.01)
            await stream.done_writing()
  1. I tried to enable gRPS keepalive while forming insecure_channel. It didn't have desired effect.
  2. Subsequently I tried calling channel_ready() inside the tight loop during queue empty, was expecting to throw some exception and come out of that loop, but it didn't work.
async with grpc.aio.insecure_channel('127.0.0.1:51234',
                  options = [
                      ('grpc.keepalive_time_ms', 60000),
                      ('grpc.keepalive_timeout_ms', 600000),
                      ('grpc.keepalive_permit_without_calls', 1),
                      ('grpc.http2.max_pings_without_data', 0),
                      ('grpc.http2.min_time_between_pings_ms', 10000),
                      ('grpc.http2.min_ping_interval_without_data_ms', 60000),
                      ('grpc.max_connection_age_ms', 2147483647),
                  ]) as channel:

Upvotes: 0

Views: 1012

Answers (1)

paul
paul

Reputation: 1

I was able to solve it using channel.get_state(). Below is the code snippet

if queue.empty():
                    if channel.get_state() != grpc.ChannelConnectivity.READY:
                        break
                    time.sleep(5)

Upvotes: 0

Related Questions