How to debug connection issue: Connecting to SSM VPC Endpoint from Lambda Function in VPC

Question

I have a lambda function in a VPC that should be able to connect to SSM via a VPC endpoint. In fact, I have had this working in another VPC (the default), but just can't get it to work in a new VPC that I have built with CloudFormation. In addition, I am able to connect to S3 via a VPC endpoint for the same VPC:

• The subnets that Lambda is being deployed in, are associated with the SSM VPC Endpoint.
• The VPC endpoint is associated with a security group that accepts incoming requests on port 443 from the application layer security group (which is associated with the lambda function).
• The Lambda function's application layer security group has an outgoing rule that allows it to broadcast anywhere 0.0.0.0/0
• The NACL for the subnets allows all incoming and outgoing traffic.

• The lambda function's role has permission to execute all SSM actions on all resources (just a temporary situation to rule out iam permissions)

• I have tried console.log debugging, and it shows the lambda function timing out after it tries to get info from SSM.

• I have tried using flow logs, but I can see no connections, or any data flowing to and from SSM.
• I have compared the 2 VPC's (the one in which it works and the one in which it doesn't) and I can't see any differences in rules.

Can anyone suggest how I might debug this?

Thanks

Answer 1

Finally found the issue and thought I'd post this just in case anyone else has the same issue.

The VPC Endpoints needed to have private dns enabled.

When creating the VPC endpoint manually via the console, the Enable Private DNS Name checkbox is checked by default.

However, when creating the VPC endpoint using CloudFormation, the PrivateDnsEnabled property needs to be explicitly set to true.