Unable to delete a k8s namespace

Question

I wanted to delete a k8s namespace but I get this error back and still didn't realize how to resolve it. Can anyone please help?

Error from server (InternalError): Internal error occurred: failed calling webhook "namespace.validators.kubedb.com": the server is currently unable to handle the request

Answer 1

With the info provided in the question, the error is pointing that there is a validation webhook configured in validators namespace and this webhook is not responding or missing.

Error from server (InternalError): Internal error occurred: failed calling webhook "namespace.validators.kubedb.com": the server is currently unable to handle the request

You can check the validation wehbhook by running:

kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io

From the below output, it is clear that your service do not have any endpoint(pods) to forward the requests.

kubectl get ep -n validators
No resources found in validators namespace.

To fix this, you need to check the validation webhook and fix or delete it.

As OP provided the output of endpoints in the validator namespace, there is no endpoint associated with the service. In such case, the behavior seems to be expected. To help the OP to debug the issue, here is a simple demo showing the relation of service/pods/endpoints.

// following is the pod that will be exposed by the service.

kubectl get pod my-pod -owide --show-labels
NAME     READY   STATUS    RESTARTS   AGE   IP            NODE            NOMINATED NODE   READINESS GATES   LABELS
my-pod   1/1     Running   0          22s   10.233.72.6   kube-worker-3   <none>           <none>            app=nginx

// note that, there is no service and endpoint associated with the pod is present

k get svc -l app=nginx
No resources found in default namespace.

k get ep -l app=nginx
No resources found in default namespace.

// exposing the pod by a service called my-pod-svc

k expose pod my-pod --name my-pod-svc --port 80
service/my-pod-svc exposed

//now note the service, endpoint and their IPs.

k get svc -l app=nginx
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
my-pod-svc   ClusterIP   10.233.26.82   <none>        80/TCP    21s
    
k get ep -l app=nginx
NAME         ENDPOINTS        AGE
my-pod-svc   10.233.72.6:80   7s

// trying to connect to the pod via service(my-pod-svc) IP

kubectl run tmp-shell --restart=Never --rm -i --tty --image busybox -- wget -nv 10.233.26.82 -o -
Connecting to 10.233.26.82 (10.233.26.82:80)
saving to 'index.html'
index.html           100% |********************************|   615  0:00:00 ETA
'index.html' saved
pod "tmp-shell" deleted

Now deleting the pod(my-pod) that is exposed by the service my-pod-svc. This also means, deleting the endpoint of the service.

 k delete  pod my-pod --force --grace-period 0
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "my-pod" force deleted

Checking the endpoint status now:

k get ep -l app=nginx
NAME         ENDPOINTS   AGE
my-pod-svc   <none>      11m

Now trying to connect to the my-pod-svc service:

kubectl run tmp-shell --restart=Never --rm -i --tty --image busybox -- wget -nv 10.233.26.82 -o -
If you don't see a command prompt, try pressing enter.
wget: can't connect to remote host (10.233.26.82): Connection refused
pod "tmp-shell" deleted
pod default/tmp-shell terminated (Error)