user630702
Reputation: 3167
Kubernetes pod stuck in pending state and no events
There aren't any other pods deployed. I have 2 nodes with zero pods. Its a t3.small instance type.
Is there any other commands to debug and see why it is still stuck in pending state?
kubectl get pods
NAME READY STATUS RESTARTS AGE
mysql-6cc489dcd5-5jc8t 0/1 Pending 0 91s
kubectl describe pod mysql-6cc489dcd5-5jc8t
Name: mysql-6cc489dcd5-5jc8t
Namespace: default
Priority: 0
Node: <none>
Labels: app=mysql
pod-template-hash=6cc489dcd5
Annotations: kubernetes.io/psp: eks.privileged
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/mysql-6cc489dcd5
Containers:
mysql:
Image: mysql:5.6
Port: 3306/TCP
Host Port: 0/TCP
Environment:
MYSQL_ROOT_PASSWORD: dbpassword11
Mounts:
/docker-entrypoint-initdb.d from usermanagement-dbcreation-script (rw)
/var/lib/mysql from mysql-persistent-storage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-nlx4p (ro)
Volumes:
mysql-persistent-storage:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: ebs-mysql-pv-claim
ReadOnly: false
usermanagement-dbcreation-script:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: usermanagement-dbcreation-script
Optional: false
default-token-nlx4p:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-nlx4p
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
Deployments shows that it has scaled and nothing wrong there.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 5m34s deployment-controller Scaled up replica set mysql-6cc489dcd5 to 1
cat 01-storage-class.yml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
cat 02-persistent-volume-claim.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ebs-mysql-pv-claim
spec:
accessModes:
- ReadWriteOnce
storageClassName: ebs-sc # Metadata name of the storage class
resources:
requests:
storage: 4Gi
cat 03-UserManagement-ConfigMap.yml
apiVersion: v1
kind: ConfigMap
metadata:
name: usermanagement-dbcreation-script
data:
mysql_usermgt.sql: |-
DROP DATABASE IF EXISTS usermgmt;
CREATE DATABASE usermgt;
cat 04-mysql-deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql
spec:
replicas: 1
selector:
matchLabels:
app: mysql
strategy:
type: Recreate
template:
metadata:
name: mysql-deployment
labels:
app: mysql
spec:
containers:
- name: mysql
image: mysql:5.6
env:
- name: MYSQL_ROOT_PASSWORD
value: dbpassword11
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage # name from the volumes listed below
mountPath: /var/lib/mysql
- name: usermanagement-dbcreation-script # name from the volumes listed below
mountPath: /docker-entrypoint-initdb.d
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: ebs-mysql-pv-claim
- name: usermanagement-dbcreation-script
configMap:
name: usermanagement-dbcreation-script
cat 05-mysql-clusterip-service.yml
apiVersion: v1
kind: Service
metadata:
name: mysql
spec:
#type: # Defaults to cluster IP service
selector:
app: mysql
ports:
- name: mysql
port: 3306 # If name isn't provided, port can start as a list `- port: 3306`
#targetPort # Defaults to containerPort
clusterIP: None # Use POD IP directly
kubectl describe nodes
Name: ip-192-168-14-238.ap-south-1.compute.internal
Roles: <none>
Labels: alpha.eksctl.io/cluster-name=eksdemo1
alpha.eksctl.io/nodegroup-name=eksdemo1-ng-public1
beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=t2.small
beta.kubernetes.io/os=linux
eks.amazonaws.com/capacityType=ON_DEMAND
eks.amazonaws.com/nodegroup=eksdemo1-ng-public1
eks.amazonaws.com/nodegroup-image=ami-01cfab2ec59e55c25
eks.amazonaws.com/sourceLaunchTemplateId=lt-02527e7be817e66a5
eks.amazonaws.com/sourceLaunchTemplateVersion=1
failure-domain.beta.kubernetes.io/region=ap-south-1
failure-domain.beta.kubernetes.io/zone=ap-south-1a
kubernetes.io/arch=amd64
kubernetes.io/hostname=ip-192-168-14-238.ap-south-1.compute.internal
kubernetes.io/os=linux
node.kubernetes.io/instance-type=t2.small
topology.ebs.csi.aws.com/zone=ap-south-1a
topology.kubernetes.io/region=ap-south-1
topology.kubernetes.io/zone=ap-south-1a
Annotations: csi.volume.kubernetes.io/nodeid: {"ebs.csi.aws.com":"i-0b3699bcac4e70d31"}
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Wed, 25 Nov 2020 05:05:47 -0500
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: ip-192-168-14-238.ap-south-1.compute.internal
AcquireTime: <unset>
RenewTime: Thu, 26 Nov 2020 01:41:17 -0500
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Thu, 26 Nov 2020 01:40:40 -0500 Wed, 25 Nov 2020 05:05:47 -0500 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 26 Nov 2020 01:40:40 -0500 Wed, 25 Nov 2020 05:05:47 -0500 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 26 Nov 2020 01:40:40 -0500 Wed, 25 Nov 2020 05:05:47 -0500 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 26 Nov 2020 01:40:40 -0500 Wed, 25 Nov 2020 05:06:17 -0500 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 192.168.14.238
ExternalIP: 15.206.94.127
Hostname: ip-192-168-14-238.ap-south-1.compute.internal
InternalDNS: ip-192-168-14-238.ap-south-1.compute.internal
ExternalDNS: ec2-15-206-94-127.ap-south-1.compute.amazonaws.com
Capacity:
attachable-volumes-aws-ebs: 39
cpu: 1
ephemeral-storage: 20959212Ki
hugepages-2Mi: 0
memory: 2039136Ki
pods: 11
Allocatable:
attachable-volumes-aws-ebs: 39
cpu: 940m
ephemeral-storage: 18242267924
hugepages-2Mi: 0
memory: 1551712Ki
pods: 11
System Info:
Machine ID: 10f1040dde9f47289c7e10ea3fb9f040
System UUID: EC2C067A-2E0E-8A49-76C1-7F5FFA16E7ED
Boot ID: d2638886-d5b3-4fa6-a15b-e95e046397f5
Kernel Version: 4.14.203-156.332.amzn2.x86_64
OS Image: Amazon Linux 2
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://19.3.6
Kubelet Version: v1.18.9-eks-d1db3c
Kube-Proxy Version: v1.18.9-eks-d1db3c
ProviderID: aws:///ap-south-1a/i-0b3699bcac4e70d31
Non-terminated Pods: (5 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system aws-node-zn9v5 10m (1%) 0 (0%) 0 (0%) 0 (0%) 20h
kube-system coredns-6b4cdc67b4-pzrl7 100m (10%) 0 (0%) 70Mi (4%) 170Mi (11%) 20h
kube-system ebs-csi-controller-7bf8974dc8-cth9n 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20h
kube-system ebs-csi-node-2vr6j 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20h
kube-system kube-proxy-dd4wd 100m (10%) 0 (0%) 0 (0%) 0 (0%) 20h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 210m (22%) 0 (0%)
memory 70Mi (4%) 170Mi (11%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
attachable-volumes-aws-ebs 0 0
Events: <none>
Name: ip-192-168-57-231.ap-south-1.compute.internal
Roles: <none>
Labels: alpha.eksctl.io/cluster-name=eksdemo1
alpha.eksctl.io/nodegroup-name=eksdemo1-ng-public1
beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=t2.small
beta.kubernetes.io/os=linux
eks.amazonaws.com/capacityType=ON_DEMAND
eks.amazonaws.com/nodegroup=eksdemo1-ng-public1
eks.amazonaws.com/nodegroup-image=ami-01cfab2ec59e55c25
eks.amazonaws.com/sourceLaunchTemplateId=lt-02527e7be817e66a5
eks.amazonaws.com/sourceLaunchTemplateVersion=1
failure-domain.beta.kubernetes.io/region=ap-south-1
failure-domain.beta.kubernetes.io/zone=ap-south-1b
kubernetes.io/arch=amd64
kubernetes.io/hostname=ip-192-168-57-231.ap-south-1.compute.internal
kubernetes.io/os=linux
node.kubernetes.io/instance-type=t2.small
topology.ebs.csi.aws.com/zone=ap-south-1b
topology.kubernetes.io/region=ap-south-1
topology.kubernetes.io/zone=ap-south-1b
Annotations: csi.volume.kubernetes.io/nodeid: {"ebs.csi.aws.com":"i-0317d2c406ad6034d"}
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Wed, 25 Nov 2020 05:05:49 -0500
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: ip-192-168-57-231.ap-south-1.compute.internal
AcquireTime: <unset>
RenewTime: Thu, 26 Nov 2020 01:41:17 -0500
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Thu, 26 Nov 2020 01:40:39 -0500 Wed, 25 Nov 2020 05:05:49 -0500 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 26 Nov 2020 01:40:39 -0500 Wed, 25 Nov 2020 05:05:49 -0500 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 26 Nov 2020 01:40:39 -0500 Wed, 25 Nov 2020 05:05:49 -0500 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 26 Nov 2020 01:40:39 -0500 Wed, 25 Nov 2020 05:06:20 -0500 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 192.168.57.231
ExternalIP: 52.66.246.122
Hostname: ip-192-168-57-231.ap-south-1.compute.internal
InternalDNS: ip-192-168-57-231.ap-south-1.compute.internal
ExternalDNS: ec2-52-66-246-122.ap-south-1.compute.amazonaws.com
Capacity:
attachable-volumes-aws-ebs: 39
cpu: 1
ephemeral-storage: 20959212Ki
hugepages-2Mi: 0
memory: 2039136Ki
pods: 11
Allocatable:
attachable-volumes-aws-ebs: 39
cpu: 940m
ephemeral-storage: 18242267924
hugepages-2Mi: 0
memory: 1551712Ki
pods: 11
System Info:
Machine ID: a14b97ab355f485294e0e6bd736775af
System UUID: EC2D6C5A-497D-E6A4-F235-CF3055FCB546
Boot ID: c37c79e9-4bb1-4ffb-810f-80a4405cb403
Kernel Version: 4.14.203-156.332.amzn2.x86_64
OS Image: Amazon Linux 2
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://19.3.6
Kubelet Version: v1.18.9-eks-d1db3c
Kube-Proxy Version: v1.18.9-eks-d1db3c
ProviderID: aws:///ap-south-1b/i-0317d2c406ad6034d
Non-terminated Pods: (5 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system aws-node-qnjzl 10m (1%) 0 (0%) 0 (0%) 0 (0%) 20h
kube-system coredns-6b4cdc67b4-whszf 100m (10%) 0 (0%) 70Mi (4%) 170Mi (11%) 20h
kube-system ebs-csi-controller-7bf8974dc8-cs6p8 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20h
kube-system ebs-csi-node-2qwjq 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20h
kube-system kube-proxy-pkxc9 100m (10%) 0 (0%) 0 (0%) 0 (0%) 20h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 210m (22%) 0 (0%)
memory 70Mi (4%) 170Mi (11%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
attachable-volumes-aws-ebs 0 0
Events: <none>
Upvotes: 9
Views: 17091
Answers (6)
Jay Perez
Reputation: 1
Is the AWS EBS volume attached to the instance? May a look in the console can help you out. Also if can share kubectl get pvc,pv.
To add to this... Looks like you are on EKS, if you are using AWS EBS you will need to install the Amazon EBS CSI driver to be able to use AWS EBS for persistentVolumes.
Upvotes: 0
taitelman
Reputation: 700
On top of the standard reason (resource limits , tolerations, volumes and a like) another possible root cause: the deployment uses non default scheduler. look up for schedulerName field and its value . when a deployment/sts uses some custom scheduler it might not honor the K8s event logging mechanism.
In managed clusters you don't always have read access to the kube-system scheduler logs .
usually kubectl get event -n [namespace of the problematic pod] should include the scheduler logs. kubectl describe pod will also emit that info in the end.
Upvotes: 3
Aaron
Reputation: 3419
In my case for some reason my cluster had been set to be cordoned which means no nodes could be scheduled. In minikube running kubectl uncordon minikube solved my problem of pods being stuck in pending. I figured this out by running kubectl get nodes and noticed the status said it could not schedule.
Upvotes: 1
Zambozo
Reputation: 486
Is the AWS EBS volume attached to the instance? May a look in the console can help you out. Also if can share kubectl get pvc,pv.
Upvotes: 4
Fritz Duchardt
Reputation: 11940
In this setup, I would first suspect something wrong with the persistent storage. Do a describe on the Persistent Volume Claim and Storage Class to see whether they are alright.
Also, if this is an EKS cluster, you can use the preinstalled EBS storage class.
Upvotes: 3
Jeff Woodard
Reputation: 647
kubectl get nodes -o wide
or
kubectl describe nodes
should give you some info about the nodes in your cluster. It could perhaps be a resource issue. I don't see any limits or requests around CPU/memory in your podspec. I'd try adding some and ensure that the minimum recommended value falls in line with what is available on your node.
Upvotes: 1
Related Questions
- Pod in Kubernetes always in pending state
- kubernetes pod status always "pending"
- Pod staying in Pending state
- Kubernetes Pod Stuck in Pending Without Indicating Any Reason
- EKS pod stuck in "Pending" state in Fargate deployment?
- Pod stuck in Pending state when trying to schedule it on AWS Fargate
- Kubernetes pod pending when a new volume is attached (EKS)
- pod stuck in pending state after kubectl apply?
- pod hangs in Pending state
- Kubernetes pod stuck in waiting state