CODEWITHSUNDEEP
pythonasynchronousgrpcbidirectionalgrpc-python
doca
doca

Reputation: 1548

GRPC bidirectional stream closes after 1 minute without messages

I am opening a GRPC bidirectional stream with a server (python3.8 specifically). After I get some data from the server, I have to do a time consuming task (it takes about 3 minutes). While doing this, I am keeping the channel open to send the results to the server as soon as the task is done

Exactly 1 minute after the server's last message, my connection closes. And I get this error

<AioRpcError of RPC that terminated with:
    status = StatusCode.INTERNAL
    details = "Received RST_STREAM with error code 1"
    debug_error_string = "{"created":"@1625250019.779494905","description":"Error received from peer ipv4:3.101.44.139:443","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"Received RST_STREAM with error code 1","grpc_status":13}"
>

To me it seems like some timeout. I am a beginner to grpc. I referred to this article and gave my channel the client options as specified in the article with every number close to 5mins. But it did not change anything. Can someone please help me understand what is going on here and how I should solve this?

TL;DR

My connection to the server is an async secure channel. Please note that my two streams are independant which is why I have to do async

channel=grpc.aio.secure_channel(address, credentials=creds, options=options)

I have two coroutines, one is waiting for the long running task to be excecuted, other receiving and sensing messages on the bi directional stream

def parse_response(resp):
    # collect data from server response
    # make `task_available=True` when all data is received

async def stream_handler():
    stub = TaskServiceStub(channel)
    req_gen = get_requests()  # An async generator for request messages
    stream: grpc.aio.StreamStreamCall = stub.Task(req_gen)
    
    async for resp in stream.__aiter__():
        parse_response(resp)

async def task_handler():
    if task_available:
        do_task()

async def main():
    await asyncio.gather(stream_handler(), task_handler())

Upvotes: 3

Views: 2849

Answers (1)

doca
doca

Reputation: 1548

Adding a read timeout to the server nginx configuration solved the issue

nginx.ingress.kubernetes.io/server-snippet: |
    client_header_timeout 3h;
    client_body_timeout 3h;
    grpc_read_timeout 3h;
    grpc_send_timeout 3h;

Upvotes: 3

Related Questions