CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaamazon-vpcaws-nat-gatewaysecretsmanager
Joe
Joe

Reputation: 422

Lambda function can't access Secrets Manager

I wrote a lambda function to access a database so the first step is to get secrets from AWS Secrets Manager. I have a private VPC as well as subnets, NAT Gateway, and security group associated with the lambda function. I also have secretsmanager.Secret.grantRead(lambda_exec_role) so the lambda should have access to Secrets Manager.

For some reason when I test it in API Gateway, I got "errno": "ETIMEDOUT" and "code": "NetworkingError" in CloudWatch. And from the printed log I had in the API, getting secrets was failed.

I also tried to add a VPC endpoint for Secrets Manager as in here, but still got the same error.

Appreciated if anyone here could help me with this or give some hints.

Many thanks!

Upvotes: 18

Views: 25250

Answers (6)

andrhamm
andrhamm

Reputation: 4411

In case anybody else arrives at this question when troubleshooting issues with the Secrets Manager lambda extension I wanted to share my experience when implementing using the Nodejs18.x runtime (brand new at this time). I was taking advantage of the new top-level await from this new runtime and hoped to retrieve my secret from SM on init (outside the handler). All my attempts failed and I was receiving a response from the extension

not ready to serve traffic, please wait

Moving the http call (to the extension's port) inside of my handler function immediately fixed the issue.

Upvotes: 6

Btibert3
Btibert3

Reputation: 40136

To chime in on this, my solution required two steps

Upvotes: 0

Victor
Victor

Reputation: 14583

By the nature of the error - network timeout - I would assume you couldn't get your secrets even if you had permissions.

Since you're dealing with a Lambda inside a VPC, please note there are two ways of placing your Lambda in a VPC:

  • in a public subnet: in this case, your Lambda CANNOT have access to the internet, so your only option would be to configure a VPC endpoint for the Secrets Manager, or
  • in a private subnet: in this case, your private subnet can have route 0.0.0.0/0 traffic to a NAT Gateway, then to an Internet Gateway (through a public subnet). This gives you the option to access Secrets Manager through its public endpoint (i.e. through the internet), or, again, through a VPC endpoint.

Upvotes: 2

Frida Schenker
Frida Schenker

Reputation: 1509

I also had a similar problem and what worked for me is to add kms:Decryptto the IAM Policy. Because according to the official AWS documentation:

If the secret is encrypted using a customer-managed key instead of the Amazon Web Services managed key aws/secretsmanager , then you also need kms:Decrypt permissions for that key. https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.get_secret_value

The IAM policy attached to Lambda should then look something like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecrets"
            ],
            "Resource": "*"
        }
    ]
}

You might want to limit Resources to something more specific based on your requirements. Hope it also works for you.

Upvotes: 1

Scott
Scott

Reputation: 47

Here's how I got it working in serverless.yml.

AWS Reference: https://docs.aws.amazon.com/secretsmanager/latest/userguide/vpc-endpoint-overview.html#vpc-endpoint

The following yml appears under the path resources.Resources.YourVPCEndpointNameHere:

# Provides access from the VPC to Secrets Manager
# See https://docs.aws.amazon.com/secretsmanager/latest/userguide/vpc-endpoint-overview.html#vpc-endpoint
Type: AWS::EC2::VPCEndpoint
Properties:
  VpcEndpointType: Interface
  ServiceName: com.amazonaws.#{AWS::Region}.secretsmanager
  PrivateDnsEnabled: true

  # Reference your VPC here
  VpcId: !Ref EC2VPC

  # Reference your subnet ids here
  SubnetIds:
    - !Ref EC2SubnetA
    - !Ref EC2SubnetB
    - !Ref EC2SubnetC

  # Reference your security group(s) here
  SecurityGroupIds:
    - !Ref EC2SecurityGroup

Tip: You'll need the plugin serverless-pseudo-parameters to make #{AWS::Region} work.

Upvotes: -1

Alexandre Hamon
Alexandre Hamon

Reputation: 1402

I had trouve with a lambda getting secret content too.

They are several things you can try :

#1 Make sure you have permission to get the secret value, I'll give you mine for a working configuration :

  • Allow:secretsmanager:GetSecretValue on your secret
  • Allow:secretsmanager:DescribeSecret on your secret
  • Allow:secretsmanager:ListSecrets on all ressources

#2 I had trouble too with my VPC and subnets. If misconfugred, you won't be able to call Secret Manager API.

  • Switch to no VPC for your lambda and check if you can get your secret OK. If it works then it means you have a problem with your VPC/subnet configuration.
  • Check your subnet configuration :
    • On public subnet, you can configure a specific endpoint for secret manager, though I couldn't make it work, don't know why.
    • On private subnet, you nea to configure a NAT Gateway to be able to call for Secret Manager API.

Hoping it could help someone someday. :)

Upvotes: 16

Related Questions