Why I am getting `Endpoint request timed out` error when AWS SecretsManager is implemented?

Question

I am developing a REST API with AWS Lambda, API Gateway, RDS (MySQL). I am using Node.js.

This is my normal AWS Lambda code, calling database table to get the data and send it over the REST API.

const mysql = require('mysql');
const con = mysql.createConnection({
  host     : "****.****.****.rds.amazonaws.com",
  user     : "****",
  password : "****",
  port     : 3306,
  database : "****"
});

exports.getAllRoles = (event, context, callback) => {
  // allows for using callbacks as finish/error-handlers
  context.callbackWaitsForEmptyEventLoop = false;
  const sql = "select * from role";
  con.query(sql, function (err, result) {
    if (err) throw err;

    var response = {
        "statusCode": 200,
        "headers": {
            "Content-Type": "application/json"
        },
        "body": JSON.stringify(result),
        "isBase64Encoded": false
    };
    callback(null, response)
  });
};

This is the same code, but now with AWS SecretsManager

const mysql = require('mysql');


// Load the AWS SDK
var AWS = require('aws-sdk'),
    region = "us-east-1",
    secretName = "test-secret",
    secret,
    decodedBinarySecret;

// Create a Secrets Manager client
var client = new AWS.SecretsManager({
    region: region
});


exports.getAllRoles = (event, context, callback) => {

    client.getSecretValue({
        SecretId: secretName
    }, function (err, data) {
        if (err) {
            throw err;
        } else {
            // Decrypts secret using the associated KMS CMK.
            // Depending on whether the secret is a string or binary, one of these fields will be populated.
            if ('SecretString' in data) {
                secret = data.SecretString;
            } else {
                let buff = new Buffer(data.SecretBinary, 'base64');
                decodedBinarySecret = buff.toString('ascii');
            }
        }
        // Your code goes here. 
        const secretObj = JSON.parse(secret);

        //Create MySQL Connection
        const con = mysql.createConnection({
            host     : secretObj.host,
            user     : secretObj.user,
            password : secretObj.password,
            port     : secretObj.port,
            database : secretObj.database
          });

        // allows for using callbacks as finish/error-handlers
        context.callbackWaitsForEmptyEventLoop = false;
        const sql = "select * from role";
        con.query(sql, function (err, result) {
            if (err) throw err;

            var response = {
                "statusCode": 200,
                "headers": {
                    "Content-Type": "application/json"
                },
                "body": JSON.stringify(result),
                "isBase64Encoded": false
            };
            callback(null, response)
        });
    });    

};

This is my cloud formation file

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  aaaa-restapi

  Sample SAM Template for aaaa-restapi
  
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 100
    VpcConfig:
        SecurityGroupIds:
          - sg-4424242424
        SubnetIds:
          - subnet-424242424242

Resources:
  
  GetAllRolesFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: aaaa-restapi/
      Handler: role-getall.getAllRoles
      Runtime: nodejs14.x
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /role/getall
            Method: get
  

  LambdaRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Path: /
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: root
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - ec2:DescribeNetworkInterfaces
                  - ec2:CreateNetworkInterface
                  - ec2:DeleteNetworkInterface
                  - ec2:DescribeInstances
                  - ec2:AttachNetworkInterface
                Resource: '*'

My code without the AWS SecretsManager is working fine when deployed.

However the code with AWS SecretsManager is giving timeout error when deployed and tested the API Call with POSTMAN. I just call

https://*****.****-api.****-1.amazonaws.com/Prod/role/getall

I just see "message": "Endpoint request timed out" and nothing else, even in the console. The same code is working fine when executed with sam invoke local GetAllRolesFunction. Also fine when tested with sam local start-api. The issue only arises when uploaded to AWS and the API Call is made.

What is happening here?

Answer 1

The error message "Endpoint request timed out" from API Gateway suggests that your API-invoked Lambda function took longer than 29 seconds to complete.

I am going to guess that your Lambda function is running in a VPC and you have correctly set up network routing to your (private) RDS database but you don't have a network route to the Secrets Manager service endpoint, hence that request times out. If so, you'd need a VPC Endpoint for Secrets Manager or a NAT (or NAT Gateway) with routing to the public internet.