Reputation: 3828

Namespace "stuck" as Terminating. How do I remove it?

I've had a "stuck" namespace that I deleted showing in this eternal "terminating" status.

Upvotes: 329

Answers (30)

Asher Manangan

Reputation: 90

kubectl get apiservice|grep False

v1beta1.metrics.k8s.io kube-system/metrics-server False (MissingEndpoints) 377d

I did renable this and it works successfully

Upvotes: 0

Jonas_Hess

Reputation: 2018

This script deletes all Kubernetes resources blocking the deletion of a specified namespace. It first removes the finalizers of each resource, then deletes the resource itself. Finally, it deletes the specified namespace.

#!/bin/bash
#
# This script deletes all Kubernetes resources blocking the deletion of a specified namespace.
# It first removes the finalizers of each resource, then deletes the resource itself.
# Finally, it deletes the specified namespace.
#
# Usage: ./delete_namespace_resources.sh [OPTIONS]
# Options:
#   -n, --namespace <terminating-namespace>   Namespace to be deleted
#   -h, --help                               Display this help and exit
#

set -euo pipefail

# log messages
log() {
    local message=$1
    echo "[INFO] $message"
}

delete_finalizers() {
    local resource_name=$1
    local namespace=$2

    # Get the resource with finalizers removed
    log "Deleting finalizers of $resource_name"
    kubectl patch "$resource_name" -n "$namespace" --type='json' -p='[{"op": "remove", "path": "/metadata/finalizers"}]'
}

delete_resource() {
    local resource_name=$1
    local namespace=$2

    # Delete the resource
    log "Deleting resource: $resource_name in namespace: $namespace"
    kubectl delete "$resource_name" -n "$namespace" --ignore-not-found
}

delete_blocking_resources() {
    local terminating_namespace=$1

    # Get all Kubernetes resources in the namespace
    resources=$(kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n "$terminating_namespace" | tail -n +2)

    # Loop through each resource
    while read -r resource; do
        local resource_name=$(echo "$resource" | awk '{print $1}')

        # Delete finalizers of the resource
        delete_finalizers "$resource_name" "$terminating_namespace"

        # Delete the resource itself
        delete_resource "$resource_name" "$terminating_namespace"
    done <<< "$resources"
}

display_help() {
    cat <<EOF
Usage: $0 [OPTIONS]
Options:
  -n, --namespace <terminating-namespace>   Namespace to be deleted
  -h, --help                               Display this help and exit
EOF
}

# Parse command line options
while [[ $# -gt 0 ]]; do
    key="$1"

    case $key in
        -n|--namespace)
            TERMINATING_NAMESPACE="$2"
            shift
            shift
            ;;
        -h|--help)
            display_help
            exit 0
            ;;
        *)
            echo "Unknown option: $1" >&2
            display_help
            exit 1
            ;;
    esac
done

# Check if terminating namespace is provided
if [ -z "${TERMINATING_NAMESPACE:-}" ]; then
    echo "Error: Terminating namespace not provided." >&2
    display_help
    exit 1
fi

# Delete resources blocking namespace deletion
delete_blocking_resources "$TERMINATING_NAMESPACE"

# Delete the namespace
log "Deleting namespace $TERMINATING_NAMESPACE"
kubectl delete namespace "$TERMINATING_NAMESPACE"
log "Namespace $TERMINATING_NAMESPACE successfully deleted"

Upvotes: 1

bgsuello

Reputation: 712

For anyone looking for few commands for later version of Kubernetes, this helped me.

NAMESPACE=mynamespace
kubectl get namespace $NAMESPACE -o json | sed 's/"kubernetes"//' | kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f -

Tested in Kubernetes v1.24.1

Note: the command replaces all kubernetes strings in the json, please use it with care

Upvotes: 14

Sascha

Reputation: 159

My attempt with a script at kubectl version v1.26.3 and jq:

remove_finalizers.sh

#!/usr/bin/bash

set -x

function rmfin() {
  for e in $(kubectl get $1 -o name); do
    kubectl get $e -o json | jq '.metadata.finalizers = []' | kubectl replace $e -f -
  done
}

rmfin $*

Execution

$ sh remove_finalizers ns

Upvotes: 1

Harsh Manvar

Reputation: 30198

Single line command

kubectl patch ns <Namespace_to_delete> -p '{"metadata":{"finalizers":null}}'

Option : 2

If Patching not working you check the error if it's like in status

kubectl get ns <namespace-name>

Discovery failed for some groups, 1 failing: unable to retrieve the 22 complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently 23 unable to handle the request

You get the error details or API details which is failing for you, list available apiservice

kubectl get apiservice

Look for ones the AVAILABLE is False

kubectl delete apiservice <apiservice-name>

Example

kubectl delete apiservice metrics.k8s.io/v1beta1

and run above patch command again or wait for while namespace will get removed.

Read article for more : https://medium.com/@harsh.manvar111/kubernetes-namespace-stuck-on-terminating-state-25d0cda8e3ff

Extra - Simple trick

You can edit namespace on console only kubectl edit <namespace name> remove/delete "Kubernetes" from inside the finalizer section(Should be like "finalizers": [ ]) and press enter or save/apply changes.

in one step also you can do it.

Trick : 1

kubectl get namespace annoying-namespace-to-delete -o json > tmp.json
then edit tmp.json and remove "kubernetes" from Finalizers
Open another terminal Run command kubectl proxy and run below Curl

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json https://localhost:8001/api/v1/namespaces/<NAMESPACE NAME TO DELETE>/finalize

and it should delete your namespace.

Step by step guide

Start the proxy using command :

kubectl proxy

kubectl proxy & Starting to serve on 127.0.0.1:8001

find namespace

kubectl get ns

{Your namespace name} Terminating 1d

put it in file

kubectl get namespace {Your namespace name} -o json > tmp.json

edit the file tmp.json and remove the finalizers

}, "spec": { "finalizers": [ "kubernetes" ] },

after editing it should look like this

}, "spec": { "finalizers": [ ] },

we are almost there simply now run the curl with updating namespace value in it

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/{Your namespace name}/finalize

and it's gone

Upvotes: 80

Jordan

Reputation: 565

1. Using Curl Command

Issue Mentioned: https://amalgjose.com/2021/07/28/how-to-manually-delete-a-kubernetes-namespace-stuck-in-terminating-state/

export NAMESPACE=<specifice-namespace>
kubectl get namespace $NAMESPACE -o json > tempfile.json

Edit the JSON file and remove all values from spec.finalizers

Save it and then apply this command on separate tab (Must be open in separate Tab)

kubectl proxy

And run this command on same tab:

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tempfile.json http://127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize

Check namespace if terminating namespace is removed or not

kubectl get namespaces

2. Using Kubectl Command

Issue Mentioned: https://aws.amazon.com/premiumsupport/knowledge-center/eks- terminated-namespaces/

Save a JSON file similar to the following:

export NAMESPACE=<specific-namespace>
kubectl get namespace $NAMESPACE -o json > tempfile.json

Edit the JSON file and remove all values from spec.finalizers
To apply the changes, run a command similar to the following:

kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f ./tempfile.json

Verify that the terminating namespace is removed:

kubectl get namespaces

Upvotes: 20

Christian Altamirano Ayala

Reputation: 1320

In my case the problem was caused by a custom metrics.

To know what is causing the issue, just run this command:

kubectl api-resources | grep -i false

That should give you which api resources are causing the problem. Once identified just delete it:

kubectl delete apiservice v1beta1.custom.metrics.k8s.io

Once deleted, the namespace should disappear.

Upvotes: 14

Deoxyseia

Reputation: 1387

I had similar problem with metrics-server, after to execute kubectl get namespace <NAMESPACE-HERE> -o json, I found this message:

"Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1?timeout=32s\": Unauthorized") has prevented the request from succeeding

I had installed an old version of metrics-server on this cluster. I only removed these resources. As it was install with:

kubectl apply -f metrics-server-0.3.7/deploy/1.8+/ --dry-run -o yaml | kubectl apply -f -

This command removed all of them:

kubectl apply -f metrics-server-0.3.7/deploy/1.8+/ --dry-run -o yaml | kubectl delete -f -

Finally, the namespace disappeared after some seconds.

I assume, if you have a more recent versions, you can delete with:

1. YAML installation (here):

kubectl delete -f https://github.com/kubernetes-sigs/metrics-server/releases/download/<VERSION-HERE>/components.yaml

2. Helm chart installation (here)

helm uninstall metrics-server

Don't forget to install this again with the correct version.

Upvotes: 0

Kamafeather

Reputation: 9855

Delete all the resources listed by:

kubectl delete -n YOURNAMESPACE —-all

Use kubectl delete -n YOURNAMESPACE <resource> <id> or (if you copy paste from the above output) kubectl delete -n YOURNAMESPACE <resource>/<id>, for each resource that you see listed there.

You can also do it at once kubectl delete -n YOURNAMESPACE <resource>/<id1> <resource>/<id2> <resource2>/<id3> <resource2>/<id4> <resource3>/<id5> etc..

Probably you tried to remove resources but they are getting recreated because of the deployment or replicaset resource, preventing the namespace from freeing up depending resources and from being cleaned up.

Upvotes: -3

DrGecko

Reputation: 899

here is a (yet another) solution. This uses jq to remove the finalisers block from the json, and does not require kubectl proxy:

namespaceToDelete=blah

kubectl get namespace "$namespaceToDelete" -o json \
  | jq 'del(.spec.finalizers)' \
  | kubectl replace --raw /api/v1/namespaces/$namespaceToDelete/finalize -f -

Upvotes: 26

Mohammed

Reputation: 757

Editing NS yaml manually didn't work for me, no error was thrown on editing but changes did not take effect.

This worked for me:

In one session:

kubectl proxy

in another shell:

kubectl get ns <rouge-ns> -o json | jq '.spec.finalizers=[]' | curl -X PUT http://localhost:8001/api/v1/namespaces/<rouge-ns>/finalize -H "Content-Type: application/json" --data @-

source: https://virtual-simon.co.uk/vsphere-kubernetes-force-deleting-stuck-terminating-namespaces-and-contexts/

Upvotes: 1

аlex

Reputation: 5698

I tried 3-5 options to remove ns, but only this one works for me.

This sh file will remove all namespaces with Terminating status

$ vi force-delete-namespaces.sh

$ chmod +x force-delete-namespaces.sh

$ ./force-delete-namespaces.sh

#!/usr/bin/env bash

set -e
set -o pipefail

kubectl proxy &
proxy_pid="$!"
trap 'kill "$proxy_pid"' EXIT

for ns in $(kubectl get namespace --field-selector=status.phase=Terminating --output=jsonpath="{.items[*].metadata.name}"); do
    echo "Removing finalizers from namespace '$ns'..."
    curl -H "Content-Type: application/json" -X PUT "127.0.0.1:8001/api/v1/namespaces/$ns/finalize" -d @- \
        < <(kubectl get namespace "$ns" --output=json | jq '.spec = { "finalizers": [] }')

    echo
    echo "Force-deleting namespace '$ns'..."
    kubectl delete namespace "$ns" --force --grace-period=0 --ignore-not-found=true
done

Upvotes: 11

Mohammad Ravanbakhsh

Reputation: 3096

Solution:

Use command below without any changes. it works like a charm.

NS=`kubectl get ns |grep Terminating | awk 'NR==1 {print $1}'` && kubectl get namespace "$NS" -o json   | tr -d "\n" | sed "s/\"finalizers\": \[[^]]\+\]/\"finalizers\": []/"   | kubectl replace --raw /api/v1/namespaces/$NS/finalize -f -

Enjoy

Upvotes: 120

alex li

Reputation: 734

I write simple script to delete your stucking namespace based on @Shreyangi Saxena 's solution.

cat > delete_stuck_ns.sh << "EOF"
#!/usr/bin/env bash

function delete_namespace () {
    echo "Deleting namespace $1"
    kubectl get namespace $1 -o json > tmp.json
    sed -i 's/"kubernetes"//g' tmp.json
    kubectl replace --raw "/api/v1/namespaces/$1/finalize" -f ./tmp.json
    rm ./tmp.json
}

TERMINATING_NS=$(kubectl get ns | awk '$2=="Terminating" {print $1}')

for ns in $TERMINATING_NS
do
    delete_namespace $ns
done
EOF

chmod +x delete_stuck_ns.sh

This Script can detect all namespaces in Terminating state, and delete it.

PS:

This may not work in MacOS, cause the native sed in macos is not compatible with GNU sed.

you may need install GNU sed in your MacOS, refer to this answer.
Please confirm that you can access your kubernetes cluster through command kubectl.
Has been tested on kubernetes version v1.15.3

Update

I found a easier solution:

kubectl patch RESOURCE NAME -p '{"metadata":{"finalizers":[]}}' --type=merge

Upvotes: 23

AnonymousX

Reputation: 1018

kubectl edit namespace ${stucked_namespace}

Then delete finalizers in vi mode and save.

It worked in my case.

Upvotes: 0

Rotem jackoby

Reputation: 22218

Debugging a similar issue.

Two important things to consider:

1 ) Think twice before deleting finalizers from your namespace because there might be resources that you wouldn't want to automatically delete or at least understand what was deleted for troubleshooting.

2 ) Commands like kubectl api-resources --verbs=list might not give you resources that were created by external crds.

In my case:

I viewed my namespace real state (that was stuck on Terminating) with kubectl edit ns <ns-name> and under status -> conditions I saw that some external crds that I installed were failed to be deleted because they add a finalizers defined:

 - lastTransitionTime: "2021-06-14T11:14:47Z"
    message: 'Some content in the namespace has finalizers remaining: finalizer.stackinstall.crossplane.io
      in 1 resource instances, finalizer.stacks.crossplane.io in 1 resource instances'
    reason: SomeFinalizersRemain
    status: "True"
    type: NamespaceFinalizersRemaining

Upvotes: 1

Kiran

Reputation: 311

Edit: It is not recommended to remove finalizers. Correct approach would be:

Delete all the resources in the namespace.

Github issue link

My usual workspace is a small k8s cluster which I frequently destroy and rebuild it back, and that's why removing finalizers method works for me.

Original answer: I usually run into same problem.

This is what I do

kubectl get ns your-namespace -o json > ns-without-finalizers.json

Edit ns-without-finalizers.json. replace all finalizers with empty array.

Run kubectl proxy ( usually run it on another terminal )

Then curl this command

curl -X PUT http://localhost:8001/api/v1/namespaces/your-namespace/finalize -H "Content-Type: application/json" --data @ns-without-finalizers.json

Upvotes: 6

Ela_murugan

Reputation: 201

Please try with below command:

kubectl patch ns <your_namespace> -p '{"metadata":{"finalizers":null}}'

Upvotes: 12

Naveen Gopalakrishna

Reputation: 219

Replace ambassador with your namespace

Check if the namespace is stuck

kubectl get ns ambassador

NAME         STATUS        AGE
ambassador   Terminating   110d

This is stuck from a long time

Open a admin terminal/cmd prompt or powershell and run

kubectl proxy

This will start a local web server: Starting to serve on 127.0.0.1:8001

Open another terminal and run

kubectl get ns ambassador -o json >tmp.json

edit the tmp.json using vi or nano

from this

{
"apiVersion": "v1",
"kind": "Namespace",
"metadata": {
    "annotations": {
        "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"name\":\"ambassador\"}}\n"
    },
    "creationTimestamp": "2021-01-07T18:23:28Z",
    "deletionTimestamp": "2021-04-28T06:43:41Z",
    "name": "ambassador",
    "resourceVersion": "14572382",
    "selfLink": "/api/v1/namespaces/ambassador",
    "uid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
},
"spec": {
    "finalizers": [
        "kubernetes"
    ]
},
"status": {
    "conditions": [
        {
            "lastTransitionTime": "2021-04-28T06:43:46Z",
            "message": "Discovery failed for some groups, 3 failing: unable to retrieve the complete list of server APIs: compose.docker.com/v1alpha3: an error on the server (\"Internal Server Error: \\\"/apis/compose.docker.com/v1alpha3?timeout=32s\\\": Post https://0.0.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews: write tcp 0.0.0.0:53284-\u0026gt;0.0.0.0:443: write: broken pipe\") has prevented the request from succeeding, compose.docker.com/v1beta1: an error on the server (\"Internal Server Error: \\\"/apis/compose.docker.com/v1beta1?timeout=32s\\\": Post https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews: write tcp 0.0.0.0:5284-\u0026gt;10.96.0.1:443: write: broken pipe\") has prevented the request from succeeding, compose.docker.com/v1beta2: an error on the server (\"Internal Server Error: \\\"/apis/compose.docker.com/v1beta2?timeout=32s\\\": Post https://0.0.0.0:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews: write tcp 1.1.1.1:2284-\u0026gt;0.0.0.0:443: write: broken pipe\") has prevented the request from succeeding",
            "reason": "DiscoveryFailed",
            "status": "True",
            "type": "NamespaceDeletionDiscoveryFailure"
        },
        {
            "lastTransitionTime": "2021-04-28T06:43:49Z",
            "message": "All legacy kube types successfully parsed",
            "reason": "ParsedGroupVersions",
            "status": "False",
            "type": "NamespaceDeletionGroupVersionParsingFailure"
        },
        {
            "lastTransitionTime": "2021-04-28T06:43:49Z",
            "message": "All content successfully deleted",
            "reason": "ContentDeleted",
            "status": "False",
            "type": "NamespaceDeletionContentFailure"
        }
    ],
    "phase": "Terminating"
}

}

    {
  "apiVersion": "v1",
  "kind": "Namespace",
  "metadata": {
    "annotations": {
      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"name\":\"ambassador\"}}\n"
    },
    "creationTimestamp": "2021-01-07T18:23:28Z",
    "deletionTimestamp": "2021-04-28T06:43:41Z",
    "name": "ambassador",
    "resourceVersion": "14572382",
    "selfLink": "/api/v1/namespaces/ambassador",
    "uid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  },
  "spec": {
    "finalizers": []
  }
}

by deleting status and kubernetes inside finalizers

Now use the command and replace ambassador with your namespace

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/ambassador/finalize

you will see another json like before then run

then run the command

 kubectl get ns ambassador
Error from server (NotFound): namespaces "ambassador" not found

If it still says terminating or any other error make sure you format your json in a proper way and try the steps again.

Upvotes: 9

imriss

Reputation: 1981

If the namespace stuck in Terminating while the resources in that namespace have been already deleted, you can patch the finalizers of the namespace before deleting it:

kubectl patch ns ns_to_be_deleted -p '{"metadata":{"finalizers":null}}';

then

kubectl delete ns ns_to_be_deleted;

Edit:

Please check @Antonio Gomez Alvarado's Answer first. The root cause could be the metrics server that mentioned in that answer.

Upvotes: 5

Luke

Reputation: 2474

Forcefully deleting the namespace or removing finalizers is definitely not the way to go since it could leave resources registered to a non existing namespace.

This is often fine but then one day you won't be able to create a resource because it is still dangling somewhere.

The upcoming Kubernetes version 1.16 should give more insights into namespaces finalizers, for now I would rely on identification strategies. A cool script which tries to automate these is: https://github.com/thyarles/knsk

However it works across all namespaces and it could be dangerous. The solution it s based on is: https://github.com/kubernetes/kubernetes/issues/60807#issuecomment-524772920

tl;dr

Checking if any apiservice is unavailable and hence doesn't serve its resources: kubectl get apiservice|grep False
Finding all resources that still exist via kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n $your-ns-to-delete

(credit: https://github.com/kubernetes/kubernetes/issues/60807#issuecomment-524772920)

Upvotes: 19

Jossef Harush Kadouri

Reputation: 34257

I've written a one-liner Python3 script based on the common answers here. This script removes the finalizers in the problematic namespace.

python3 -c "namespace='<my-namespace>';import atexit,subprocess,json,requests,sys;proxy_process = subprocess.Popen(['kubectl', 'proxy']);atexit.register(proxy_process.kill);p = subprocess.Popen(['kubectl', 'get', 'namespace', namespace, '-o', 'json'], stdout=subprocess.PIPE);p.wait();data = json.load(p.stdout);data['spec']['finalizers'] = [];requests.put('http://127.0.0.1:8001/api/v1/namespaces/{}/finalize'.format(namespace), json=data).raise_for_status()"

💡 rename namespace='<my-namespace>' with your namespace. e.g. namespace='trust'

Full script: https://gist.github.com/jossef/a563f8651ec52ad03a243dec539b333d

Upvotes: 27

Abhi Gadroo

Reputation: 210

Something similar happened to me in my case it was pv & pvc , which I forcefully removed by setting finalizers to null. Check if you could do similar with ns

kubectl patch pvc <pvc-name> -p '{"metadata":{"finalizers":null}}'

For namespaces it'd be

kubectl patch ns <ns-name> -p '{"spec":{"finalizers":null}}'

Upvotes: 1

dbustosp

Reputation: 4478

I loved this answer extracted from here It is just 2 commands.

In one terminal:

kubectl proxy

In another terminal:

kubectl get ns delete-me -o json | \
  jq '.spec.finalizers=[]' | \
  curl -X PUT http://localhost:8001/api/v1/namespaces/delete-me/finalize -H "Content-Type: application/json" --data @-

Upvotes: 70

Shreyangi Saxena

Reputation: 1057

Need to remove the finalizer for kubernetes.

Step 1:

kubectl get namespace <YOUR_NAMESPACE> -o json > <YOUR_NAMESPACE>.json

remove kubernetes from finalizers array which is under spec

Step 2:

kubectl replace --raw "/api/v1/namespaces/<YOUR_NAMESPACE>/finalize" -f ./<YOUR_NAMESPACE>.json

Step 3:

kubectl get namespace

You can see that the annoying namespace is gone.

Upvotes: 104

wind_surfer

Reputation: 39

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json 127.0.0.1:8001/k8s/clusters/c-mzplp/api/v1/namespaces/rook-ceph/finalize

This worked for me, the namespace is gone.

Detailed explanation can be found in the link https://github.com/rook/rook/blob/master/Documentation/ceph-teardown.md.

This happened when I interrupted kubernetes installation(Armory Minnaker). Then I proceeded to delete the namespace and reinstall it. I was stuck with pod in terminating status due to finalizers. I got the namespace into tmp.json, removed finalizers from tmp.json file and did the curl command. Once I get past this issue, I used scripts for uninstalling the cluster to remove the residues and did a reinstallation.

Upvotes: 0

rhozet

Reputation: 572

The simplest and most easiest way of doing this is copying this bash script

#!/bin/bash

###############################################################################
# Copyright (c) 2018 Red Hat Inc
#
# See the NOTICE file(s) distributed with this work for additional
# information regarding copyright ownership.
#
# This program and the accompanying materials are made available under the
# terms of the Eclipse Public License 2.0 which is available at
# http://www.eclipse.org/legal/epl-2.0
#
# SPDX-License-Identifier: EPL-2.0
###############################################################################

set -eo pipefail

die() { echo "$*" 1>&2 ; exit 1; }

need() {
    which "$1" &>/dev/null || die "Binary '$1' is missing but required"
}

# checking pre-reqs

need "jq"
need "curl"
need "kubectl"

PROJECT="$1"
shift

test -n "$PROJECT" || die "Missing arguments: kill-ns <namespace>"

kubectl proxy &>/dev/null &
PROXY_PID=$!
killproxy () {
    kill $PROXY_PID
}
trap killproxy EXIT

sleep 1 # give the proxy a second

kubectl get namespace "$PROJECT" -o json | jq 'del(.spec.finalizers[] | select("kubernetes"))' | curl -s -k -H "Content-Type: application/json" -X PUT -o /dev/null --data-binary @- http://localhost:8001/api/v1/namespaces/$PROJECT/finalize && echo "Killed namespace: $PROJECT"

# proxy will get killed by the trap

Add the above code in the deletenamepsace.sh file.

And then execute it by providing namespace as parameter(linkerd is the namespace i wanted to delete here)

➜ kubectl get namespaces
linkerd           Terminating   11d

➜ sh deletenamepsace.sh linkerd
Killed namespace: linkerd

➜ kubectl get namespaces

The above tip has worked for me.

Honestly i think kubectl delete namespace mynamespace --grace-period=0 --force is not at all worth trying.

Special Thanks to Jens Reimann! I think this script should be incorporated in kubectl commands.

Upvotes: -1

Saurav Malani

Reputation: 191

Run kubectl get apiservice

For the above command you will find an apiservice with Available Flag=Flase.

So, just delete that apiservice using kubectl delete apiservice <apiservice name>

After doing this, the namespace with terminating status will disappear.

Upvotes: 19

prince

Reputation: 73

Run the following command to view the namespaces that are stuck in the Terminating state:

kubectl get namespaces
Select a terminating namespace and view the contents of the namespace to find out the finalizer. Run the following command:

kubectl get namespace -o yaml
Your YAML contents might resemble the following output:

        apiVersion: v1
        kind: Namespace
        metadata:
           creationTimestamp: 2019-12-25T17:38:32Z
           deletionTimestamp: 2019-12-25T17:51:34Z
           name: <terminating-namespace>
           resourceVersion: "4779875"
           selfLink: /api/v1/namespaces/<terminating-namespace>
           uid: ******-****-****-****-fa1dfgerz5
         spec:
           finalizers:
           - kubernetes
         status:
           phase: Terminating

Run the following command to create a temporary JSON file:

kubectl get namespace -o json >tmp.json
Edit your tmp.json file. Remove the kubernetes value from the finalizers field and save the file. Output would be like:

    {
        "apiVersion": "v1",
        "kind": "Namespace",
        "metadata": {
            "creationTimestamp": "2018-11-19T18:48:30Z",
            "deletionTimestamp": "2018-11-19T18:59:36Z",
            "name": "<terminating-namespace>",
            "resourceVersion": "1385077",
            "selfLink": "/api/v1/namespaces/<terminating-namespace>",
            "uid": "b50c9ea4-ec2b-11e8-a0be-fa163eeb47a5"
        },
        "spec": {
        },

        "status": {
            "phase": "Terminating"
        }
    }

To set a temporary proxy IP and port, run the following command. Be sure to keep your terminal window open until you delete the stuck namespace:

kubectl proxy
Your proxy IP and port might resemble the following output:

Starting to serve on 127.0.0.1:8001
From a new terminal window, make an API call with your temporary proxy IP and port:

  curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/your_terminating_namespace/finalize

Your output would be like:

    {
       "kind": "Namespace",
       "apiVersion": "v1",
       "metadata": {
         "name": "<terminating-namespace>",
         "selfLink": "/api/v1/namespaces/<terminating-namespace>/finalize",
         "uid": "b50c9ea4-ec2b-11e8-a0be-fa163eeb47a5",
         "resourceVersion": "1602981",
         "creationTimestamp": "2018-11-19T18:48:30Z",
         "deletionTimestamp": "2018-11-19T18:59:36Z"
       },
       "spec": {

       },
       "status": {
         "phase": "Terminating"
       }
    }

The finalizer parameter is removed. Now verify that the terminating namespace is removed, run the following command:

kubectl get namespaces

Upvotes: 6

teoincontatto

Reputation: 2219

As mentioned before in this thread there is another way to terminate a namespace using API not exposed by kubectl by using a modern version of kubectl where kubectl replace --raw is available (not sure from which version). This way you will not have to spawn a kubectl proxy process and avoid dependency with curl (that in some environment like busybox is not available). In the hope that this will help someone else I left this here:

kubectl get namespace "stucked-namespace" -o json \
  | tr -d "\n" | sed "s/\"finalizers\": \[[^]]\+\]/\"finalizers\": []/" \
  | kubectl replace --raw /api/v1/namespaces/stucked-namespace/finalize -f -

Upvotes: 199

Namespace &quot;stuck&quot; as Terminating. How do I remove it?

Answers (30)

remove_finalizers.sh

Execution

Single line command

Update

Related Questions

Namespace "stuck" as Terminating. How do I remove it?