Kubernetes Namespaces stuck in Terminating status

Question

We have one cluster where it seems that namespaces never want to be deleted completely and now can't re-create custom-metrics namespace to be able to collect custom metrics to properly setup HPA. I fully understand that I can create another namespace with all custom-metrics resources, but a little concerned with the overall health of the cluster, given that the namespaces get stuck in "Terminating" state

$ kubectl get ns
NAME             STATUS        AGE
cert-manager     Active        14d
custom-metrics   Terminating   7d
default          Active        222d
nfs-share        Active        15d
ingress-nginx    Active        103d
kube-public      Active        222d
kube-system      Active        222d
lb               Terminating   4d
monitoring       Terminating   6d
production       Active        221d

I already tried to export namespaces to JSON, delete finalizers and re-create using edited JSON files. also tried to kubectl edit ns custom-metrics and delete "- kubernetes" finalizer. all to no avail.

does anyone have any other recommendations on how else I can try to destroy these "stuck" namespaces"

curl to https://master-ip/api/v1/namespace/...../finalize doesn't seem to work on Google Kubernetes Engine for me, I'm assuming these operations are not allowed on GKE cluster

Trying things like doesn't work as well:

$ kubectl delete ns custom-metrics --grace-period=0 --force

warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (Conflict): Operation cannot be fulfilled on namespaces "custom-metrics": The system is ensuring all content is removed from this namespace. Upon completion, this namespace will automatically be purged by the system.

and there're no resources listed in this namespaces at all: kubectl get all -n custom-metrics or looping through all api-resources in this namespace shows no resources exist at all: kubectl api-resources --namespaced=true -o name | xargs -n 1 kubectl get -n custom-metrics

Answer 1

For All Terminating Namespaces:

kubectl get ns | grep Terminating | awk '{print $1}' | while read ns; do 
  kubectl get namespace "$ns" -o json | jq '.spec.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/$ns/finalize" -f -; 
done

This will process all namespaces stuck in the "Terminating" state and remove their finalizers.

Answer 2

No need to modify anything.

No need to use curl to talk to the api-server.

$ export NAMESPACE=cert-manager
$ kubectl get ns ${NAMESPACE} -o json | jq '.spec.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/${NAMESPACE}/finalize" -f -

Now, make sure the namespace is gone:

$ kubectl get ns ${NAMESPACE}
Error from server (NotFound): namespaces "cert-manager" not found

Answer 3

In my case the solution was:

kubectl delete -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Indeed, when setting up metrics, I tried to use metrics-server, but didn't work. So I used https://bitnami.com/stack/prometheus-operator/helm instead, but didn't remove metrics-server.
By doing

kubectl api-resources --verbs=list --namespaced -o name \                                                                         
  | xargs -n 1 kubectl get --show-kind --ignore-not-found -n devops-87

then
kubectl describe APIService v1beta1.metrics.k8s.io

I realized that the problem was this unused metrics-server.

Answer 4

It's very simple. you can simply follow 2 easy ways,

1. Remove/commend finalizers under metadata by using kubectl edit ns <namespace-name>

2. if there is no Finalizers under meta data only under spec then you can follow the below,

   export NAMESPACE=name-space then

   kubectl get namespace $NAMESPACE -o json | tr -d "\n" | sed "s/\"finalizers\": \[[^]]\+\]/\"finalizers\": []/" | kubectl replace --raw /api/v1/namespaces/$NAMESPACE/finalize -f -

Answer 5

This solution worked for me,

kubectl patch namespace cattle-system -p '{"metadata":{"finalizers":[]}}' --type='merge' -n cattle-system

kubectl delete namespace cattle-system --grace-period=0 --force

"cattle-system" is the namespace to delete

Answer 6

I did something similar to rahul.tripathi except the curl did not work for me - I followed https://medium.com/@craignewtondev/how-to-fix-kubernetes-namespace-deleting-stuck-in-terminating-state-5ed75792647e which does the following:

NAMESPACE=
kubectl get namespace $NAMESPACE -o json > $NAMESPACE.json
sed -i -e 's/"kubernetes"//' $NAMESPACE.json
kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f ./$NAMESPACE.json

Voila! Namespace is deleted

Update: One-liner version of this solution (requires jq)

NAMESPACE= ; kubectl get namespace $NAMESPACE -o json | jq 'del(.spec.finalizers[0])' | kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f -

Update #2: Terraform version

resource "kubernetes_namespace" "this" {
  for_each = toset( var.namespaces )
  metadata {
    name = each.key
  }
  provisioner "local-exec" {
    when = destroy
    command    = "nohup ${path.module}/namespace-finalizer.sh ${each.key} 2>&1 &"
  }
}

namespace-finalizer.sh

sleep 30; kubectl get namespace $1 && kubectl get namespace $1 -o json | jq 'del(.spec.finalizers[0])' | kubectl replace --raw "/api/v1/namespaces/$1/finalize" -f -

Answer 7

Had the problem deleting a namespace:

kubectl delete namespaces "localkube-ns" 

Error from server (Conflict):
Operation cannot be fulfilled on namespaces "localkube-ns": 
The system is ensuring all content is removed from this namespace.  
Upon completion, this namespace will automatically be purged by the system.

After several long minutes, the problem disapeard.

The namespaced was probably long to delete after a problem that did generate a lot of Evicted pods.

This command seems faster: kubectl delete all --all --namespace=<NAMESPACE_NAME>

Answer 8

• 1. Get state current:

kubectl get namespace <namespace-to-delete> -o json > tmp.json

Obs.: Replace

• 2. edit tmp.json and remove "kubernetes" from "spec": { "finalizers":['kubernetes']}

Ex.: "spec": { "finalizers":[ ]}

• 3. Alter the namespace and delete:

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json https://<kubernetes-cluster-ip>/api/v1/namespaces/<namespace-to-delete>/finalize -H "Authorization: Bearer <token-Account>"

Obs.: Replace namespace-to-delete, token-Account and token-Account

Answer 9

The only solution that worked for me was:

1. kubectl get namespace annoying-namespace-to-delete -o json > tmp.json

2. edit tmp.json and remove"kubernetes" from "spec": { "finalizers":[]}

3. curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json https://kubernetes-cluster-ip/api/v1/namespaces/annoying-namespace-to-delete/finalize

and this should delete your namespace,

Answer 10

Was able to reproduce by installing a Prometheus operator from this repo and then just trying to delete a namespace.

First run:

k apply -f manifests/

That command creates monitoring namespace, a bunch of namespaced resources like deployments and configmaps as well as non-namespaced ones like roles etc.

Then imperatively deleting the namespace:

k delete ns monitoring

with an idea the Kubernetes will delete all the corresponding resources. As a result all objects in the namespace were deleted however namespace itself get stuck in the Terminated state

Just to illustrate, here is a list of stray resources left after "deleting" the namespace. Those resources got deleted only after running the kubectl delete on the corresponding folder:

customresourcedefinition.apiextensions.k8s.io "podmonitors.monitoring.coreos.com" deleted
customresourcedefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" deleted
customresourcedefinition.apiextensions.k8s.io "prometheusrules.monitoring.coreos.com" deleted
customresourcedefinition.apiextensions.k8s.io "servicemonitors.monitoring.coreos.com" deleted
clusterrole.rbac.authorization.k8s.io "prometheus-operator" deleted
clusterrolebinding.rbac.authorization.k8s.io "prometheus-operator" deleted
clusterrole.rbac.authorization.k8s.io "kube-state-metrics" deleted
clusterrolebinding.rbac.authorization.k8s.io "kube-state-metrics" deleted
clusterrole.rbac.authorization.k8s.io "node-exporter" deleted
clusterrolebinding.rbac.authorization.k8s.io "node-exporter" deleted
apiservice.apiregistration.k8s.io "v1beta1.metrics.k8s.io" deleted
clusterrole.rbac.authorization.k8s.io "prometheus-adapter" deleted
clusterrole.rbac.authorization.k8s.io "system:aggregated-metrics-reader" deleted
clusterrolebinding.rbac.authorization.k8s.io "prometheus-adapter" deleted
clusterrolebinding.rbac.authorization.k8s.io "resource-metrics:system:auth-delegator" deleted
clusterrole.rbac.authorization.k8s.io "resource-metrics-server-resources" deleted
rolebinding.rbac.authorization.k8s.io "resource-metrics-auth-reader" deleted
clusterrole.rbac.authorization.k8s.io "prometheus-k8s" deleted
clusterrolebinding.rbac.authorization.k8s.io "prometheus-k8s" deleted
rolebinding.rbac.authorization.k8s.io "prometheus-k8s" deleted
role.rbac.authorization.k8s.io "prometheus-k8s" deleted

This experiment is likely proving the idea if your namespace is stuck in Terminated state there are always resources left referring it and preventing it to get deleted. The easiest (and correct) way to clean it up is using the same instrumentation as when creating it (kubectl apply, Helm etc).

Answer 11

For me, deletion with --grace-period=0 --force has never worked. Rico's answer is good, but probably you can do it without restarting your cluster.

In my case, there are ALWAYS some objects which were recreated after you have "deleted" your namespace.

To see which Kubernetes resources are and aren’t in a namespace:

kubectl api-resources --namespaced=true
kubectl api-resources --namespaced=false

What I am doing is to go through it and find all k8s objects which were in some use of that specific namespace, and delete them manually.

EDIT: Another useful command for finding objects that should be deleted:

kubectl api-resources --verbs=list --namespaced -o name \
  | xargs -n 1 kubectl get --show-kind --ignore-not-found -l <label>=<value> -n <namespace>

Answer 12

Looks like this is a known issue with people having mixed results trying a mix of different things:

• Bounce the kube-controller-manager
• Bounce all the kubelets
• Bounce the whole cluster
• kubectl delete ns <name> --grace-period=0 --force
• Patching finalizers "null" everywhere.

Some more background but at the pod level here too.