CODEWITHSUNDEEP
kuberneteskubectlkubernetes-podazure-aks
Thomas
Thomas

Reputation: 2126

AKS - incorrect Pod Status

I have an AKS Cluster with two nodepools. Node pool 1 has 3 nodes, and nodepool 2 has 1 node - all Linux VMs. I noticed that after stopping the VMs and then doing kubectl get pods, the Pods status shows "running" though the VMs are not actually running. How is this possible?

This is the command I tried: kubectl get pods -n development -o=wide

The screenshot is given below. Though VMs are not running, the Pod status shows "running". However, trying to access the app using the Public IP of the service resulted in

ERR_CONNECTION_TIMED_OUT

enter image description here

Upvotes: 0

Views: 889

Answers (2)

Karishma Tiwari - MSFT
Karishma Tiwari - MSFT

Reputation: 1555

What is actually going is related to the kubelet processes running on the nodes cannot provide their status to the Kubernetes API server. Kubernetes will always assume that your PODs are running when the nodes associated with the POD are offline. The fact that all nodes are offline, will in fact cause your POD to not be running hence not being accessible, causing the ERR_CONNECTION_TIMED_OUT

You can run kubectl get nodes to get the status of the nodes, they should show NotReady. Please check and let me know. Also, can you please provide the output for kubectl get pods -A

Upvotes: 0

Anmol Agrawal
Anmol Agrawal

Reputation: 904

Here is a full thread (https://github.com/kubernetes/kubernetes/issues/55713) on this issue. The problem here is by default the pod waits for 5 minutes before evicting to another node when the current node becomes notReady, but in this case none of the worker nodes are ready and hence pods are not getting evicted. Refer the git issue, there are some suggestions and solutions provided.

Upvotes: 2

Related Questions