AWS IAM Role for Service Account with EBS CSI driver: could not create volume in EC2: NoCredentialProviders: no valid providers in chain

Question

I have successfully installed the AWS EBS CSI driver to my EKS cluster.

This is meant to be using the "IAM Role for Service Account" technique.

I am trying to utilise the checkout example app that AWS have given here The pod will not come up (pending) and the PVC is showing this:

Name:          ebs-claim
Namespace:     test
StorageClass:  ebs-sc
Status:        Pending
Volume:        
Labels:        app=ebs-claim
               com.mylabel.contact=dl-myteam.dlonp1
Annotations:   volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
               volume.kubernetes.io/selected-node: ip-10-232-100-115.ec2.internal
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       meme-ebs
Events:
  Type     Reason              Age               From                                                                                      Message
  ----     ------              ----              ----                                                                                      -------
  Warning  ProvisioningFailed  27s               persistentvolume-controller                                                               storageclass.storage.k8s.io "ebs-sc" not found
  Normal   Provisioning        8s (x4 over 25s)  ebs.csi.aws.com_ebs-csi-controller-6dfdb77cdf-fbsbz_1760973c-09bb-43ab-b005-ffcd818447fc  External provisioner is provisioning volume for claim "test/ebs-claim"
  Warning  ProvisioningFailed  5s (x4 over 22s)  ebs.csi.aws.com_ebs-csi-controller-6dfdb77cdf-fbsbz_1760973c-09bb-43ab-b005-ffcd818447fc  failed to provision volume with StorageClass "ebs-sc": rpc error: code = Internal desc = Could not create volume "pvc-05efbff8-9506-4003-9bab-e1ce4719bc1c": could not create volume in EC2: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request

Similar to an issue I saw here, but had no answers.

Can anyone suggest things to try? Seems like the IAM role is not wired thru to the API that mounts the volume on EC2?

Answer 1

I had the same issue and fixed it by updating the Amazon EBS CSI driver IAM role as documented here

Set up driver permissions

1. Create a new IAM role for granting the AssumeRoleWithWebIdentity action

1a. Copy the following contents to a file that's named aws-ebs-csi-driver-trust-policy.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com",
          "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa"
        }
      }
    }
  ]
}

❕Note: Remember to add the audience sts.amazonaws.com for aud: sts.amazonaws.com and also to add both for aud and sub parameters in Condition section as mentioned above ↑↑

1b. Create a new IAM role amazoneks_ebs_csi_driver_role using above aws-ebs-csi-driver-trust-policy.json

aws iam create-role \
  --role-name amazoneks_ebs_csi_driver_role \
  --assume-role-policy-document file://"aws-ebs-csi-driver-trust-policy.json"

1c. Attach the AmazonEBSCSIDriverPolicy AWS managed policy to IAM role amazoneks_ebs_csi_driver_role with the following command

aws iam attach-role-policy \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --role-name amazoneks_ebs_csi_driver_role

2. [Optional] If you are using custom KMS key for encryption on your Amazon EBS volumes, customise the IAM role as needed. Read Step.4 here

❕Note: Make sure to update the commands and policies to a different name.

---

Set up EKS IAM roles for service accounts (IRSA)

Create the service account with the same name used in OIDC auth sub i.e. ebs-csi-controller-sa for above created IAM role amazoneks_ebs_csi_driver_role in your EKS cluster and make sure you add below annotation to service account:

  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/amazoneks_ebs_csi_driver_role

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ebs-csi-controller-sa
  labels:
    app.kubernetes.io/name: aws-ebs-csi-driver
  #Enable if EKS IAM roles for service accounts (IRSA) is used. See https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html for details.
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/amazoneks_ebs_csi_driver_role
EOF

❕Note: Make sure that there is NO typos in IAM role arn for annotation. ebs-csi-controller-sa SA is added to controller deployment

---

Finally restart the Amazon EBS CSI driver controller

kubectl -n kube-system rollout restart deployment/ebs-csi-controller

---

Verify that the Amazon EBS CSI driver is working

You can test the CSI driver functionality by deploying a sample application.

Read here

---

❕Note: You can also read this troubleshoot issues with my EBS volume mounts in Amazon EKS guide

Answer 2

Ensure that the created ServiceAccounts have the correct IRSA annotations.

If you are using the helm chart, and doing an upgrade from an older version, double check the location of the IRSA ServiceAccount annotation (they may have changed, had me stumped for a bit as to why things didn't work).

Answer 3

Looks like an issue with the service account that your efs csi driver is using. For example, make sure it's using the right role with the right trust policy for your EKS cluster. For example check the right annotation below:

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/name: aws-efs-csi-driver
  name: efs-csi-controller-sa
  namespace: kube-system
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/AmazonEKS_EFS_CSI_DriverRole

And the role that you are using has the right trust policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:kube-system:efs-csi-controller-sa"
        }
      }
    }
  ]
}

The instructions here are pretty clear. (It worked for me)