Kubernetes Service Selector change doesn't take effect on connected Clients

Question

I want to display a maintenance page on an application running under Kubernetes whilst a deployment is in progress, in this “maintenance” window, I backup the database and then apply schema changes and then deploy the new version.

I thought maybe what I could do is change the service selector so that it would point to a nginx container serving up a simple maintenance page whilst the deployment progressed. Once the deployment had succeeded, I would switch back the selector to point to the pods that do the actual work.

My problem with this is approach is that unless I close and reopen the browser that is currently looking at the site then I never see the maintenance page; I’m guessing the browser is keeping a connection open. The public service address doesn’t change throughout this process.

I’m testing this locally on a Docker Kubernetes installation using a type of NodePort .

Any ideas on how to get it working or am I flogging a dead horse with this approach?

Regards Lee

Answer 1

First of all make sure the content you are serving is not cached.

Second, make sure to close all open TCP connections when you shut down your pods. The steps should be as follows:

1. Change service selector to route traffic to maintenance pods
2. Gracefully shutdown running pods (this includes closing all open TCP connections)
3. Do maintenance
4. Change service selector back

As an alternative approach, you can use an ingress controller. That won't have this problem, because it doesn't maintain an open TCP connection to the pods.

Answer 2

This happens due to a combination of how browsers and k8s services work.

Browsers cache TCP connections to servers: when requesting a page they will leave the TCP connection open, and if the user later requests more pages from the same domain, the browser will reuse the already-open TCP connection to save time.

The k8s service load balancing operates at the TCP layer. When a new TCP connection is received, it will be assigned to a pod from the Service, and it will keep talking to that pod for the entire TCP connection's lifetime.

So, the issue is your browser is keeping TCP connections open to your old pods, even if you modify the service.

How can we fix this?

Non-solution #1: have the browser not cache connections. As far as I know there's no way to do this, and you don't want it anyway because it'll make your site slower. Also, HTTP caching headers have no impact on this. Browsers always cache TCP connections. A no-cache header will make the browser request the page again, but over the already-open connection.

Non-solution #2: have k8s kill TCP connections when updating the service. This is not possible and is not desirable either because this behavior is what makes "graceful shutdown / request draining" deployment strategies work. See issue.

Solution #1: Use Layer 7 (HTTP) load balancing instead of Layer 4 (TCP) load balancing, such as nginx-ingress. L7 load balancing routes traffic to pods "per HTTP request", instead of "per TCP connection", so you won't have this problem even if browsers keep TCP connections open.

Solution #2: do this from your application instead of from k8s. For example, have an "in-maintenance" DB flag, check it on every request and serve the maintenance page if it's set.

Answer 3

Here is how services in Kubernetes work, they are basically a dummy loadbalancers forwarding requests to pods in a round robin fashion, and they select which pods to forward the requests to based on the labels as you have already figured out.

Now here is how http/tcp work, I open the browser to visit your website www.example.com the tcp takes it's round of syn,ack,syn-ack and I receive the data.

In your case once I open your website I get a reply from a certain pod based on how the service routed me, and that's it, no further communication is made.

Afterwards you remove the functional pods from the service and add the maintenance page, this will be only shown to the new clients connecting to your website.

I.E if I requested your website, and then you changed all the code and restarted NGINX, if I didn't refresh I would not receive new content