AWS sam cli local invoke lambda with secrets manager port undefined

Question

I am trying to test my lambda function locally using sam local invoke. Error says UnknownEndpoint: Inaccessible host: secretsmanager.us-east-1.amazonaws.com' at port undefined' This error is being thrown from inside my lambda function code as that is were I am pull secrets from. I have tried using --region --profile options as well but no luck. For context, I am using terraform to design and deploy my infrastructure. Using SAML Authorization with Credentials file for AWS Access to our VPC environment. I have verified the region is being set correctly when SAM spins up the Lambda docker container. I have also verified that I am providing the same parameters for Lambda to identify secrets manager as the one running in the VPC version.

This only thing that I see odd is the port being undefined in console that seems that it is coming internally from the AWS SDK. Note that when I used the secrets manager terraform module that has been created by our company's cloud engineering team, I don't have to provide any port information. Hope someone can help explain this issue error.

USACCMNBSTEMD6R:balance-inquiry czl74b$ sam local invoke -t ./sam-local/template.yaml -e ./sam-local/event.json --debug
2022-01-06 17:23:29,736 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2022-01-06 17:23:29,736 | Using config file: samconfig.toml, config environment: default
2022-01-06 17:23:29,736 | Expand command line arguments to:
2022-01-06 17:23:29,736 | --template_file=/Users/czl74b/dev-js/lending-api-innovation/src/apis/sor/balance-inquiry/sam-local/template.yaml --event=./sam-local/event.json --no_event --layer_cache_basedir=/Users/czl74b/.aws-sam/layers-pkg --container_host=localhost --container_host_interface=127.0.0.1 
2022-01-06 17:23:29,736 | local invoke command is called
2022-01-06 17:23:29,743 | No Parameters detected in the template
2022-01-06 17:23:29,761 | There is no customer defined id or cdk path defined for resource BalanceInquiry, so we will use the resource logical id as the resource id
2022-01-06 17:23:29,761 | There is no customer defined id or cdk path defined for resource CommonUtils, so we will use the resource logical id as the resource id
2022-01-06 17:23:29,761 | There is no customer defined id or cdk path defined for resource NpmLibs, so we will use the resource logical id as the resource id
2022-01-06 17:23:29,761 | 3 stacks found in the template
2022-01-06 17:23:29,762 | No Parameters detected in the template
2022-01-06 17:23:29,774 | There is no customer defined id or cdk path defined for resource BalanceInquiry, so we will use the resource logical id as the resource id
2022-01-06 17:23:29,774 | There is no customer defined id or cdk path defined for resource CommonUtils, so we will use the resource logical id as the resource id
2022-01-06 17:23:29,774 | There is no customer defined id or cdk path defined for resource NpmLibs, so we will use the resource logical id as the resource id
2022-01-06 17:23:29,775 | 3 resources found in the stack 
2022-01-06 17:23:29,775 | No Parameters detected in the template
2022-01-06 17:23:29,790 | There is no customer defined id or cdk path defined for resource BalanceInquiry, so we will use the resource logical id as the resource id
2022-01-06 17:23:29,790 | There is no customer defined id or cdk path defined for resource CommonUtils, so we will use the resource logical id as the resource id
2022-01-06 17:23:29,790 | There is no customer defined id or cdk path defined for resource NpmLibs, so we will use the resource logical id as the resource id
2022-01-06 17:23:29,790 | No Parameters detected in the template
2022-01-06 17:23:29,802 | There is no customer defined id or cdk path defined for resource BalanceInquiry, so we will use the resource logical id as the resource id
2022-01-06 17:23:29,802 | There is no customer defined id or cdk path defined for resource CommonUtils, so we will use the resource logical id as the resource id
2022-01-06 17:23:29,803 | There is no customer defined id or cdk path defined for resource NpmLibs, so we will use the resource logical id as the resource id
2022-01-06 17:23:29,803 | --base-dir is not presented, adjusting uri ../../../../common-utils relative to /Users/czl74b/dev-js/lending-api-innovation/src/apis/sor/balance-inquiry/sam-local/template.yaml
2022-01-06 17:23:29,803 | No Parameters detected in the template
2022-01-06 17:23:29,815 | There is no customer defined id or cdk path defined for resource BalanceInquiry, so we will use the resource logical id as the resource id
2022-01-06 17:23:29,815 | There is no customer defined id or cdk path defined for resource CommonUtils, so we will use the resource logical id as the resource id
2022-01-06 17:23:29,815 | There is no customer defined id or cdk path defined for resource NpmLibs, so we will use the resource logical id as the resource id
2022-01-06 17:23:29,815 | --base-dir is not presented, adjusting uri ../../../../npm-libs relative to /Users/czl74b/dev-js/lending-api-innovation/src/apis/sor/balance-inquiry/sam-local/template.yaml
2022-01-06 17:23:29,815 | Found Serverless function with name='BalanceInquiry' and CodeUri='../'
2022-01-06 17:23:29,816 | --base-dir is not presented, adjusting uri ../ relative to /Users/czl74b/dev-js/lending-api-innovation/src/apis/sor/balance-inquiry/sam-local/template.yaml
2022-01-06 17:23:29,840 | Found one Lambda function with name 'BalanceInquiry'
2022-01-06 17:23:29,840 | Invoking main.handler (nodejs14.x)
2022-01-06 17:23:29,840 | Environment variables overrides data is standard format
2022-01-06 17:23:29,840 | Loading AWS credentials from session with profile 'None'
2022-01-06 17:23:29,850 | Resolving code path. Cwd=/Users/czl74b/dev-js/lending-api-innovation/src/apis/sor/balance-inquiry/sam-local, CodeUri=/Users/czl74b/dev-js/lending-api-innovation/src/apis/sor/balance-inquiry
2022-01-06 17:23:29,850 | Resolved absolute path to code is /Users/czl74b/dev-js/lending-api-innovation/src/apis/sor/balance-inquiry
2022-01-06 17:23:29,850 | Code /Users/czl74b/dev-js/lending-api-innovation/src/apis/sor/balance-inquiry is not a zip/jar file
2022-01-06 17:23:29,850 | Code /Users/czl74b/dev-js/lending-api-innovation/src/common-utils is not a zip/jar file
2022-01-06 17:23:29,850 | Code /Users/czl74b/dev-js/lending-api-innovation/src/npm-libs is not a zip/jar file
2022-01-06 17:23:29,850 | CommonUtils is a local Layer in the template
2022-01-06 17:23:29,850 | Resolving code path. Cwd=/Users/czl74b/dev-js/lending-api-innovation/src/apis/sor/balance-inquiry/sam-local, CodeUri=/Users/czl74b/dev-js/lending-api-innovation/src/common-utils
2022-01-06 17:23:29,850 | NpmLibs is a local Layer in the template
2022-01-06 17:23:29,850 | Resolving code path. Cwd=/Users/czl74b/dev-js/lending-api-innovation/src/apis/sor/balance-inquiry/sam-local, CodeUri=/Users/czl74b/dev-js/lending-api-innovation/src/npm-libs
2022-01-06 17:23:29,851 | arn:aws:lambda:us-east-1:027255383542:layer:AWS-AppConfig-Extension:55 is already cached. Skipping download
Building image................................
2022-01-06 17:23:41,146 | Skip pulling image and use local one: samcli/lambda:nodejs14.x-x86_64-d5b52b0afc3579e405e95c7df.

2022-01-06 17:23:41,146 | Mounting /Users/czl74b/dev-js/lending-api-innovation/src/apis/sor/balance-inquiry as /var/task:ro,delegated inside runtime container
2022-01-06 17:23:41,598 | Starting a timer for 3 seconds for function 'BalanceInquiry'
START RequestId: 3b9f7abb-02d1-46e8-8b6b-321f9e5467ed Version: $LATEST
2022-01-07T00:23:43.539Z    3b9f7abb-02d1-46e8-8b6b-321f9e5467ed    INFO    getSecrets :: getSecretValue Error: UnknownEndpoint: Inaccessible host: `secretsmanager.us-east-1.amazonaws.com' at port 'undefined'. This service may not be available in the `us-east-1' region.

Answer 1

SAM local invoke runs the lambda function as a docker container. If behind corporate proxy, AWS SDK from this lambda needs proxy setup to communicate with the actual AWS Services. I was able to resolve by using the proxy-agent npm module. You can read about it here.

https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/node-configuring-proxies.html

Here is how this looked like in the code.

const AWS = require('aws-sdk');

const { HTTP_PROXY, LOCAL } = process.env;

if(LOCAL === 'TRUE'){
    // lazy load proxy-agent only in LOCAL for sam local testing
    const proxy = require('proxy-agent');
    AWS.config.update({ httpOptions: { agent: proxy(HTTP_PROXY>) }});
}