Can't attach EBS volume to statefulset pod. Wrong fs type, bad option, bad superblock on /dev/XXX, missing codepage or helper program, or other error

Question

Error:

  Warning  FailedMount             108s                kubelet                  Unable to attach or mount volumes: unmounted volumes=[nexus], unattached volumes=[nexus kube-api-access-4p4gw nexus-pvc]: timed out waiting for the condition
  Warning  FailedMount             76s (x18 over 21m)  kubelet                  MountVolume.MountDevice failed for volume "nexus" : rpc error: code = Internal desc = could not format "/dev/nvme1n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/nexus/globalmount": mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t xfs -o nouuid,defaults /dev/nvme1n1 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/nexus/globalmount
Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/nexus/globalmount: wrong fs type, bad option, bad superblock on /dev/nvme1n1, missing codepage or helper program, or other error.

My resources in K8s: PVC:

$ kubectl describe pvc -n nexus nexus
Name:          nexus
Namespace:     nexus
StorageClass:  io2
Status:        Bound
Volume:        nexus
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      10000Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       nexus-0
Events:        <none>

PV:

kubectl describe pv nexus
Name:            nexus
Labels:          <none>
Annotations:     pv.kubernetes.io/bound-by-controller: yes
Finalizers:      [kubernetes.io/pv-protection external-attacher/ebs-csi-aws-com]
StorageClass:    io2
Status:          Bound
Claim:           nexus/nexus
Reclaim Policy:  Retain
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        10000Gi
Node Affinity:   <none>
Message:
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            ebs.csi.aws.com
    FSType:            xfs
    VolumeHandle:      vol-XXXXXXXXXXXXX
    ReadOnly:          false
    VolumeAttributes:  <none>
Events:                <none>

Statefulset volume sections:

      volumes:
      - name: nexus
        persistentVolumeClaim:
          claimName: nexus
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: nexus-pvc
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 20Gi
      storageClassName: aws-efs
      volumeMode: Filesystem



        volumeMounts:
        - mountPath: /nexus-data-efs
          name: nexus-pvc
        - mountPath: /nexus-data
          name: nexus

It's worth to mention that everything is working as expected when I am deleting EBS volume and keep EFS.

Also:

• I can attach volume on basic ec2 instance, but I am also getting this error: wrong fs type, bad option, bad superblock on /dev/nvme1n1, missing codepage or helper program, or other error. but this option: "-o nouuid" solves the issue.

• I am able to mount efs to statefulset

• EBS was created manually, formated to xfs type so it's not a problem with ext4 filesystem type. file -s on mounted EBS shows: "SGI XFS"

• I am using ebs csi driver

• I added IAM Role to nodes (EBS CSI Driver Role)

• EKS version 1.21

Answer 1

If the Statefulset snippet is complete, I am wondering why it is missing volumeClaimTemplates for ebs

Also, based on the output of the Persistent Volume (PV) description, it appears that the PV with the name nexus was automatically provisioned, not manually provisioned.

I think The PV specifies the io2 StorageClass, which indicates that the volume is dynamically provisioned using the EBS io2 StorageClass. Dynamic provisioning means that the PV was automatically created when a Persistent Volume Claim (PVC) was requested, and it matched the criteria defined in the StorageClass.

Can you please list the Persistent Volumes in the nexus namespace to verify

If you want to use a manually created EBS (Elastic Block Store) volume in a StatefulSet in Kubernetes, you need to follow these steps:

• You should create a Persistent Volume (PV) and specify the volumeID
• Create a Persistent Volume Claim (PVC) that will request the specific PV you created in the previous step.
• Attach PVC to the StatefulSe: specify the created PVC in the volumeClaimTemplates section. Make sure to use the correct PVC name that you created in the previous step.