Shlomi
Shlomi

Reputation: 3792

gRPC DEADLINE_EXCEEDED even that the server is up and

I have two microservices that communicate each other thru gRPC, A is the RPC client and B is the RPC server, both written in NodeJS using grpc NPM module.

Everything is working fine until, at some point in time, unexpectedly A stop being able to send requests to B, it fails because of a timeout (5s) and throw this error:

Error: Deadline Exceeded

Both microservices are Docker containers, run on AWS ECS and communicate thru AWS ELB (not ALB because it does not support HTTP2 and some other problems).

I tried to run telnet from A to the ELB of B, both from the EC2 instance and from the running ECS task (the Docker container itself) itself and it connected fine, but still, the NodeJS application in A cannot reach the NodeJS application in B using the gRPC connection.

The only way to solve it is to stop and start the ECS tasks and then A succeed to connect to B again (until the next unexpected time the same scenario is reproduced), but it's not a solution of course.

Do anyone faced with that kind of issue?

Upvotes: 10

Views: 9153

Answers (2)

jdwyah
jdwyah

Reputation: 1292

I have found that I need to create both a new stub, but also re-create the connection after some errors in order to get it to reconnect. (Also running in ECS)

Upvotes: 0

Alex Borysov
Alex Borysov

Reputation: 281

Do you use unary or streaming API? Do you set any deadline? gRPC deadline is per-stream, so in case of streaming when you set X milliseconds deadline, you'll get DEADLINE_EXCEEDED X milliseconds after you opened a stream (not send or receive any messages!). And you'll keep getting it forever for this stream, the only way to get rid of it is reopening a stream.

Upvotes: 0

Related Questions