CODEWITHSUNDEEP
kubernetesgoogle-kubernetes-engine
user140547
user140547

Reputation: 8200

GKE and Graceful node shutdown with differences concerning pods in Shutdown state

I have two GKE clusters, both running 1.20.8-gke.900 for Control Plane as well as nodes, and using preemptible nodes.

According to GKE documentation

On preemptible GKE nodes running versions 1.20 or later, the kubelet graceful node shutdown feature is enabled by default. As a result, kubelet detects preemption and gracefully terminates Pods.

However, on one of these clusters when doing kubectl get pods --all-namespaces I see many pods in SHUTDOWN state, such as

kube-system      metrics-server-v0.3.6-9c5bbf784-w8cs2                       0/2     Shutdown   0          2d20h

On the other one, there are no such pods, although both have the same config as far as I know (the only difference is that the one without those pods is a private cluster, the other public, but this should not make no difference?)

Is there any setting or config difference that I have missed?

Upvotes: 2

Views: 1329

Answers (1)

Mikolaj S.
Mikolaj S.

Reputation: 3224

Posted community wiki based on comments for better visibility.

For now, the issue with pods being in constant SHUTDOWN state on the private cluster is gone. Worth to note, that the issue occurred later also on the public cluster.

A possible cause of unexpected and random behavior is that feature "Graceful node shutdown" is in alpha state in Kubernetes v.1.20. In Kubernetes v.1.21 it is in beta state.

Upvotes: 1

Related Questions