Kubernetes - pod has unbound immediate PersistentVolumeClaims

Question

I'm using mysql Kubernetes statefulset, i mapped PVs to host directory (CentOS 8 VM) but getting " pod has unbound immediate PersistentVolumeClaims"

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql-container
spec:
  serviceName: mysql
  replicas: 1 
  selector:
    matchLabels:
      app: mysql-container
  template:
    metadata:
      labels:
        app: mysql-container
    spec:
      containers:
      - name: mysql-container
        image: mysql:dev
        imagePullPolicy: "IfNotPresent"
        envFrom:
          - secretRef:
             name: prod-secrets
        ports:
        - containerPort: 3306
         # container (pod) path
        volumeMounts:
          - name: mysql-persistent-storage
            mountPath: /var/lib/mysql
      
      
      volumes:
        - name: mysql-persistent-storage
          persistentVolumeClaim:
           claimName: mysql-pvc
 volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        
        storageClassName: localstorage
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 3Gi
        selector:
           matchLabels:
            type: local

  

Storage class is default and no events in PV

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:

  name: localstorage

provisioner: kubernetes.io/no-provisioner
volumeBindingMode: Immediate
reclaimPolicy: Delete
allowVolumeExpansion: True


kind: PersistentVolume
apiVersion: v1
metadata:
  name: mysql-01
  labels:
    type: local
spec:
  storageClassName: localstorage
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/mysql01"

---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: mysql-02
  labels:
    type: local
spec:
  storageClassName: localstorage
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/mysql02"

Storage class is default one

get sc
NAME                     PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
localstorage (default)   kubernetes.io/no-provisioner   Delete          Immediate           true                   35m

PVC also shows no events:

    Name:          data-mysql-0
    Namespace:     default
    StorageClass:  localstorage
    Status:        Pending
    Volume:        mysql-storage
    Labels:        app=mysql
    Annotations:   <none>
    Finalizers:    [kubernetes.io/pvc-protection]
    Capacity:      0
    Access Modes:
    VolumeMode:    Filesystem
    Mounted By:    mysql-0
    Events:        <none>


Name:            mysql-01
Labels:          type=local
Annotations:     kubectl.kubernetes.io/last-applied-configuration:
                   {"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"labels":{"type":"local"},"name":"mysql-01"},"spec":{"accessMode...
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    localstorage
Status:          Available
Claim:
Reclaim Policy:  Retain
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        10Gi
Node Affinity:   <none>
Message:
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /mnt/mysql01
    HostPathType:
Events:            <none>


Name:            mysql-02
Labels:          type=local
Annotations:     kubectl.kubernetes.io/last-applied-configuration:
                   {"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"labels":{"type":"local"},"name":"mysql-02"},"spec":{"accessMode...
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    localstorage
Status:          Available
Claim:
Reclaim Policy:  Retain
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        10Gi
Node Affinity:   <none>
Message:
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /mnt/mysql02
    HostPathType:
Events:            <none>

Pod is in pending state:

>       Events:
>          Type     Reason            Age                From               Message
>          ----     ------            ----               ----               -------
>          Warning  FailedScheduling  27s (x2 over 27s)  default-scheduler  error while running >"VolumeBinding" filter plugin for pod "mysql-0": pod has unbound immediate PersistentVolumeClaims

Can someone point out what else should be done here, thanks

Answer 1

The error can be caused by multiple reasons - below are a few of them.

---

Example 1

The persistentvolume-controller has failed to find a PV with a capacity greater or equal than the value specified in the PVC.

For example:

  # PVC
  resources:
     requests:
       storage: 3Gi


  # PV
  capacity:
    storage: 10Gi 

So:
If PV capacity >= PVC capacity then the PVC should be able to bound to the PV.

If not then we'll get the unbound immediate PersistentVolumeClaims error in Pod logs and no volume plugin matched name when describing the PVC.

---

Example 2

The number of PVCs exceeds the number of available PVs.

For example if only one PV has been created (or the others were deleted):

$ kubectl get pv
NAME             CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                      STORAGECLASS    REASON   AGE
mongo-local-pv   50Gi       RWO            Retain           Bound    default/mongo-persistent-storage-mongo-0   local-storage            106m

We could see that some workloads (Pods or StatefulSets) will be stuck in the Pending state:

$ kubectl get pods
NAME      READY   STATUS    RESTARTS   AGE
mongo-0   2/2     Running   0          3m38s
mongo-1   0/2     Pending   0          3m23s

$ kubectl get pvc
NAME                               STATUS    VOLUME           CAPACITY   ACCESS MODES   STORAGECLASS    AGE
mongo-persistent-storage-mongo-0   Bound     mongo-local-pv   50Gi       RWO            local-storage   80m
mongo-persistent-storage-mongo-1   Pending                                              local-storage   45m

and we will get the OP's error (unbound [immediate] PersistentVolumeClaims).

---

Example 3

If the scheduler failed to find a node with a matching label for the PV.

When using local volumes the nodeAffinity of the PV is required and should be a value of a node available in the cluster:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-mongo-pv
  .
  .
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node-which-does-not-exists  # <----- Will lead to an error

---

Example 4

A PV with the same name but a different configuration already exists in the cluster and the new PVC is created according to them.

When working with local volumes the administrator must perform a manual clean-up and set up the local volumes again each time for reuse.

(*) This local static provisioner was created to help with the PV lifecycle.

Answer 2

PersistentVolumeClaims will remain unbound indefinitely if a matching PersistentVolume does not exist. The PersistentVolume is matched with accessModes and capacity. In this case capacity the PV is 10Gi whereas PVC has capacity of 3Gi.

The capacity in the PV needs to same as in the claim i.e 3Gi to fix the unbound immediate PersistentVolumeClaims issue.