CODEWITHSUNDEEP
gokubernetesenvoyproxy
Afzal Khan
Afzal Khan

Reputation: 101

envoy proxy does not give error even if pod crashes

My 2 microservices are talking to each other via GRPC and I am using envoy proxy. Microservice A is calling Microservice B and envoy container is inside Mircoservice A's pod.

My problem is that if because of any issue Microservice B pod crashed I dont get any error at A, its just keeps trying to establish connection.

My expectation is that since B pod is crashed envoy proxy should immediately give some error that cant connect or something because it knows that connection is not possible as no pods are available.

I have a standard connection timeout at A but I want envoy to fail quick in this case.

Nothing can be done at B side since it will be already crashed and A is relying on envoy proxy.

Upvotes: 1

Views: 326

Answers (1)

Hemanth Kumar
Hemanth Kumar

Reputation: 3772

In order to know the Status or Error when the microservice B is getting failed in Envoy Proxy has a feature called Envoy Circuit Breaking. This stops sending requests to pod B as it is experiencing the issues and not able to respond. Refer to this blog by Christian Posta who has explained clearly about Circuit Breaking with Envoy Proxy and its configuration with a demo.

Envoy Proxy circuit breaking is working for the microservices whereas in order to know the Pod level crash and its alert we need to use Prometheus. Prometheus is a good fit for microservices because you just need to expose a metrics port, and don’t need to add too much complexity or run additional services. You can find templates for Pod Crashing and Pod unhealthy here configure accordingly in the pods which will give you alert when Pod crashing or pod unhealthy.

Upvotes: 2

Related Questions