External-secret error AccessDeniedException for cross AWS Account secret manager

Question

Describe the bug Trying to fetch the secret in AWS Account 1 (in which AWS EKS and ESO are installed ) from AWS Account 2 (in which the secret resists) but I faced this error:

Warning  UpdateFailed  1s (x12 over 12s)  external-secrets  AccessDeniedException: User: arn:aws:sts::AWSACCOUNTID1:assumed-role/EKSSecretsReaderRole/external-secrets-provider-aws is not authorized to perform: secretsmanager:GetSecretValue on resource: appsecret/test/new-test because no identity-based policy allows the secretsmanager:GetSecretValue action
           status code: 400,

Note: I have to mention that I can perfectly get any secret from AWS Account 1 but I can not get a secret from AWS Account 2 (cross AWS account)!!

What I did Steps to reproduce the behavior:

1. I Installed AWS EKS, and ESO (installed in external-secrets namespace)
2. Create AWS policy and Role in AWS Account 1
3. Create a secret in secret manager in AWS Account 2 with a specific policy
4. Give access to KMS on AWS Account 2 to Role of AWS Account 1
5. Create and service account ( service account name external-secrets ) in the external secret namespace with annotation of role in AWS Account1
6. Create a cluster secret store and give access to the service account of external-secret.
7. Both secrets exist in both AWS Accounts in the us-east-2 region

---

Role Trust relationships:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::AWSACCOUNT1ID:oidc-provider/oidc.eks.us-east-2.amazonaws.com/id/11111112222333333"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-2.amazonaws.com/id/11111112222333333:aud": "sts.amazonaws.com",
                    "oidc.eks.us-east-2.amazonaws.com/id/11111112222333333:sub": "system:serviceaccount:external-secrets:external-secrets"
                }
            }
        }
    ]
}

---

Polic has attached to the role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetResourcePolicy",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds"
            ],
            "Resource": [
                "arn:aws:secretsmanager:us-east-2:AWSACCOUNT1ID:secret:appsecret/dev/*",
                "arn:aws:secretsmanager:us-east-2:AWSACCOUNT1ID:secret:appsecret/stage/*",
                "arn:aws:secretsmanager:us-east-2:AWSACCOUNT2ID:secret:appsecret/test/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "kms:Decrypt",
            "Resource": [
                "arn:aws:kms:us-east-2:AWSACCOUNT1ID:key/KMSIDINAWSACCOUNT1",
                "arn:aws:kms:us-east-2:AWSACCOUNT2ID:key/KMSIDINAWSACCOUNT2"
            ]
        }
    ]
}

---

Policy has attached to secret on AWS Account 2:

{
  "Version" : "2012-10-17",
  "Statement" : [ {
    "Effect" : "Allow",
    "Principal" : {
      "AWS" :  "arn:aws:iam::AWSACCOUNT1ID:role/EKSSecretsReaderRole"
    },
    "Action" : "secretsmanager:*",
    "Resource" : "*"
  } ]
}

---

Added policy to KMS in AWS Account 2:

{
  "Sid": "AllowUseOfTheKeyFromAWSACCOUNT1IDEKSSecretsReaderRole",
  "Effect": "Allow",
  "Principal": {
      "AWS": [
          "arn:aws:iam::AWSACCOUNT1ID:role/EKSSecretsReaderRole"
      ]
  },
  "Action": [
      "kms:Decrypt",
      "kms:DescribeKey"
  ],
  "Resource": "*"
}

Expected behavior I expect that when trying to apply my external secret object to fetch my secret from the cross AWS Account and create a secret from that object's external secret.

Screenshots When I am trying to apply an external secret object it shows SecretSyncedError with this error:

Events:
  Type     Reason        Age                From              Message
  ----     ------        ----               ----              -------
Warning  UpdateFailed  1s (x12 over 12s)  external-secrets  AccessDeniedException: User: arn:aws:sts::AWSACCOUNTID1:assumed-role/EKSSecretsReaderRole/external-secrets-provider-aws is not authorized to perform: secretsmanager:GetSecretValue on resource: appsecret/test/new-test because no identity-based policy allows the secretsmanager:GetSecretValue action
           status code: 400,

Additional context

EKS version: 1.26
ESO helm chart version: v0.8.3

Even I check my policy inside a pod in the same nmaespace of external-secrets by the below command and I could get the secret but external-secrets object can not get:

root@nginxpod:/# env
AWS_DEFAULT_REGION=us-east-2
AWS_REGION=us-east-2
AWS_ROLE_ARN=arn:aws:iam::AWSACCOUNT1ID:role/EKSSecretsReaderRole


root@nginxpod:/# aws secretsmanager get-secret-value --region us-east-2 --secret-id arn:aws:secretsmanager:us-east-2:AWSACCOUNT2ID:secret:appsecret/test/new-test-5sKKXc
{
    "ARN": "arn:aws:secretsmanager:us-east-2:AWSACCOUNT2ID:secret:appsecret/test/new-test-5sKKXc",
    "Name": "appsecret/test/new-test",
    "VersionId": "7731dc19-206e-4405-84d8-a34a678c9f48",
    "SecretString": "{\"name\":\"majid\"}",
    "VersionStages": [
        "AWSCURRENT"
    ],
    "CreatedDate": "2023-09-06T18:46:08.432000+00:00"
}

In the end, I have to say again I get any secret from the same AWS account that EKS has been installed but I can not get a secret from a cross AWS Account!!

I tryied everything as I mentioned and I expect to external-secrets object create automatically my secret.

Answer 1

I solved my issue by doing below steps:

1. I added allows the role to sts:assumeRole permissions in Account1 to assume the role of account2.
2. Added this line role: arn:aws:iam::AWSACCOUNTID2:role/AllowReadSecret to the ClusterSecretStore object.

by changing the above config my External-secrets is able to create a secret.