CODEWITHSUNDEEP
nginxkuberneteslifecyclekubernetes-pod
bitstream
bitstream

Reputation: 459

How can I ensure kubernetes pods (sitting behind nginx) can complete in-flight requests before pods are terminated?

My infrastructure has pods running on k8s with nginx sitting in the middle. According to the k8s pod lifecycle documentation, if you remove a pod, 1 second later it's removed from the endpoints list so no new requests are sent to it and nginx can no longer resolve it.

Am I correct in understanding that as soon as the pod is removed from the endpoints list that it's basically non-existent despite the fact that it's likely still alive and processing the SIGTERM request?

If yes, is there a way to give it a chance to complete in-flight requests while not accepting new ones?

Upvotes: 0

Views: 1018

Answers (1)

Blender Fox
Blender Fox

Reputation: 5625

Yes you are right, as soon as pod is marked as Terminating it cannot (normally) receive traffic. If you want to ensure connections are drained, you use the preStop hook described here:

(https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination).

We use this to drain connections from our Varnish deployment before it terminates, because in our setup, they don't drain "naturally" fast enough, and connections are abruptly terminated when the pods get the SIGKILL signal.

Upvotes: 1

Related Questions