Binding statefulset to local persistent volumes - volume node affinity conflict error

Question

I have 3-node kubernetes, host names are host_1, host_2, host_3.

$ kubectl get nodes
NAME       STATUS    ROLES     AGE       VERSION
host_1     Ready     master    134d      v1.10.1
host_2     Ready     <none>    134d      v1.10.1
host_3     Ready     <none>    134d      v1.10.1

I have defined 3 local persistent volumes of size 100M, mapped to a local directory on each node. I used the following descriptor 3 times where <hostname> is one of: host_1, host_2, host_3:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-volume-<hostname>
spec:
  capacity:
    storage: 100M
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /opt/jnetx/volumes/test-volume
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - <hostname>

After applying three such yamls, I have the following:

$ kubectl get pv
NAME                 CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM    STORAGECLASS    REASON    AGE
test-volume-host_1   100M       RWO            Delete           Available            local-storage             58m
test-volume-host_2   100M       RWO            Delete           Available            local-storage             58m
test-volume-host_3   100M       RWO            Delete           Available            local-storage             58m

Now, I have a very simple container that writes to a file. The file should be located on the local persistent volume. I deploy it as a statefulset with 1 replica and map volumes via statefulset's volumeClaimTemplates:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: filewriter
spec:
  serviceName: filewriter
  ...
  replicas: 1
  template:
    spec:
      containers:
        - name: filewriter
          ...
          volumeMounts:
          - mountPath: /test/data
            name: fw-pv-claim
  volumeClaimTemplates:
  - metadata:
      name: fw-pv-claim
    spec:
      accessModes:
      - ReadWriteOnce
      storageClassName: local-storage
      resources:
        requests:
          storage: 100M

The volume claim seems to have been created ok and bound to pv on the first host:

$ kubectl get pv
NAME                 CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                              STORAGECLASS    REASON    AGE
test-volume-host_1   100M       RWO            Delete           Bound       default/fw-pv-claim-filewriter-0   local-storage             1m
test-volume-host_2   100M       RWO            Delete           Available                                      local-storage             1h
test-volume-host_3   100M       RWO            Delete           Available                                      local-storage             1h

But, the pod hangs in Pending state:

$ kubectl get pods
NAME                         READY     STATUS    RESTARTS   AGE
filewriter-0                 0/1       Pending   0          4s

If we describe, we can see the following errors:

$ kubectl describe pod filewriter-0
Name:           filewriter-0
...
Events:
  Type     Reason            Age              From               Message
  ----     ------            ----             ----               -------
  Warning  FailedScheduling  2s (x8 over 1m)  default-scheduler  0/3 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 2 node(s) had volume node affinity conflict. 

Can you help me figure out what is wrong? Why can't it just create the pod?

Answer 1

I had a very similar case to the above and observed the same symptom (node affinity conflict). In my case the issue was that I had 2 volumes attached to 2 different nodes but was trying to use them within 1 pod.

I detected this by using kubectl describe pvc name-of-pvc and noting the selected-node annotation. Once I set the pod to use volumes that were both on one node, I no longer had issues.

Answer 2

It seems that the one node where the PV is available has a taint that your StatefulSet does not have toleration for.