Kubernetes: external secrets operator error: InvalidClientTokenId: The security token included in the request is invalid

Question

I am trying out external secrets operator (ESO) by following below:

https://github.com/external-secrets/external-secrets

https://external-secrets.io/guides-getting-started/

I am using minikube and AWS secrets manager to do this (I also tried it out in k8s cluster hosted in EC2, but I get the same exact error).

I followed the steps from the links above:

1. Added the repo:

helm repo add external-secrets https://charts.external-secrets.io

2. Install it in the namespace. I'm already inside of the namespace, so I didn't create the namespace.

helm install external-secrets external-secrets/external-secrets --set installCRDs=true

3. Added k8s secret:

k create secret generic aws-credentials --from-literal=aws-access-key-id='xxx' --from-literal=aws-secret-access-key='xxx'

4. I created a secret in AWS secrets manager called test_user_1
5. kubectl apply -f secret-store.yaml

kind: SecretStore
metadata:
  name: secretstore-sample
spec:
  provider:
    aws:
      service: SecretsManager
      role: arn:aws:iam::123456789012:role/somerole
      region: us-east-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: aws-credentials
            key: aws-access-key-id
          secretAccessKeySecretRef:
            name: aws-credentials
            key: aws-secret-access-key

6. kubectl apply -f externalsecrets.yaml

apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: example
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: secretstore-sample
    kind: SecretStore
  target:
    name: secret-to-be-created
    creationPolicy: Owner
  data:
    - secretKey: user-1-username
      remoteRef:
        key: test_user_1
        property: username
    - secretKey: user-1-password
      remoteRef:
        key: test_user_1
        property: password

Then it says

externalsecret.external-secrets.io/example created

When I do

kubectl describe externalsecret.external-secrets.io/example

Below is what I get and no secret-to-be-created is created:

...
Status:
  Conditions:
    Last Transition Time:  2021-06-09T22:45:10Z
    Message:               could not get secret data from provider: key "test_user_1" from ExternalSecret "example": InvalidClientTokenId: The security token included in the request is invalid.
                           status code: 403, request id: 5a544aa0-3953-4c0d-9dab-37bde10e328b
    Reason:                SecretSyncedError
    Status:                False
    Type:                  Ready
  Refresh Time:            <nil>
Events:                    <none>

I know this role has access to aws secrets manager (I've run python scripts to access aws secrets manager from my laptop using this role). But, I have limited knowledge of k8s, so, I appreciate any help.

Answer 1

I solved the issue. It's on AWS side. I needed to create a new user and a new role. Put the newly created role in the role: section of the configmap and allow the user to assume that role by providing the aws creds for the user as environment variables in the k8s cluster.

Put this policy for the newly created role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetResourcePolicy",
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret",
        "secretsmanager:ListSecretVersionIds"
      ],
      "Resource": [
        "arn:aws:secretsmanager:us-west-2:111122223333:secret:dev-*",
      ]
    }
  ]
}

And below as the trust relationship for the role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::..."
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Below policy for the newly created user:

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::...."
    }
}