CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaterraform
208_man
208_man

Reputation: 1728

Why is aws lambda getting "AccessDeniedExceptionKMS" Error Message?

I just deployed a lambda (using Terraform from gitlab runner) to a new aws account. This pipeline deploys a lambda to another (dev/test) account without issues, but when I try to deploy to my prod account, I get the following error:

KMS access was denied...

I'm honing in on the statement, "The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access."

I have confirmed that the encryption config for the env vars are set to use default aws/lambda key instead of a customer master key. That seems to contradict the language of the error which refers to a customer master key...?

The role assumed by the lambda does have a policy which includes two kms actions:

        "Sid": "AWSKeyManagementService",

        "Action": [

            "kms:Decrypt",

            "kms:DescribeKey"

        ]

By process of elimination, I wonder if the issue is a lack on the part of the resource-based policy on the kms key. Looking in the kms keys, under aws managed, I find the aws/lambda key has the following key policy:

{
    "Version": "2012-10-17",
    "Id": "auto-awslambda",
    "Statement": [
        {
            "Sid": "Allow access through AWS Lambda for all principals in the account that are authorized to use AWS Lambda",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:CreateGrant",
                "kms:DescribeKey"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "kms:ViaService": "lambda.us-east-1.amazonaws.com",
                    "kms:CallerAccount": "REDACTED"#<-- Account where lambda deployed
                }
            }
        },
        {
            "Sid": "Allow direct access to key metadata to the account",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::REDACTED:root"#<-- Account where lambda deployed
            },
            "Action": [
                "kms:Describe*",
                "kms:Get*",
                "kms:List*",
                "kms:RevokeGrant"
            ],
            "Resource": "*"
        }
    ]
}

This is very puzzling. Any pointers appreciated!

Upvotes: 5

Views: 6768

Answers (3)

Mathieu Rey
Mathieu Rey

Reputation: 661

I got a somewhat similar error on a Lambda that used to work fine before:

Lambda was unable to decrypt the environment variables because KMS access was denied. Please check the function's KMS key settings. KMS Exception: AccessDeniedExceptionKMS Message: User: arn:aws:sts::xxx is not authorized to perform: kms:Decrypt on resource: arn:aws:kms:xxx because no resource-based policy allows the kms:Decrypt action

Turns out I recently deleted the role that the lambda was assuming, then re-created it under the same name.

I changed the lambda configuration to assume another role (so that it registers a change ; just confirming the existing value in the console did lead to redeploying the function but did not solve the problem), then back to assuming the "re-created" role, and it solved the problem.

I guess something similar happened to the OP, with Terraform probably deleting then re-creating the assumed role under the same name, not realizing this is not a no-op.

Upvotes: 2

Ant&#243;nio Pedro Leite
Ant&#243;nio Pedro Leite

Reputation: 41

Also deleting and redeploying the lambda sorted it out for me after many other tries to sort this.

Upvotes: 3

208_man
208_man

Reputation: 1728

This was solved by simply deleting the lambda and then re-running my pipeline to re-deploy it. All I can conclude is that something was corrupted in the first deployment.

Upvotes: 11

Related Questions