CODEWITHSUNDEEP
amazon-web-serviceselasticsearchamazon-ec2aws-cloudformationamazon-vpc
noncom
noncom

Reputation: 4992

Allowing Lambda in a VPC to access an Elasticsearch domain in the same VPC

I am learning to get around with Amazon services, and in particular I currently want to create a simple setup with a Cloud Formation script: a VPC with a single lambda written in JS that has an access to an Elasticsearch service within the same VPC.

Somehow I can't get it to work. All requests from the lambda to the Elasticsearch domain always time out. However, the same requests made from both the same JS code or curl (even without any additional authorizations, just curling the ES domain endpoint) from a EC2 instance running Amazon Linux 2 in the same VPC work fine and I can communicate with Elasticsearch just fine from that EC2 instance (being SSHed into it).

At the same time the lambda is capable of accessing an Aurora cluster in the VPC, so it is not the general question of the lambda being unable to access the VPC resources.

Please tell me what am I doing wrong in my Cloud Formation description of the setup? Below is the relevant extract from my Cloud Formation template and the example of the JS code that is able to access the ES service from the EC2 instance, but can't do the same thing for the lambda:

AWSTemplateFormatVersion: 2010-09-09
Description: The AWS CloudFormation tutorial
Resources:

  SomeDeploymentBucket:
    Type: 'AWS::S3::Bucket'

  AppLogGroup:
    Type: 'AWS::Logs::LogGroup'
    Properties:
      LogGroupName: /aws/lambda/some-lambda

    # ========= The Lambda Execution Role =========

  IamRoleLambdaExecution:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Policies:
        - PolicyName: !Join 
            - '-'
            - - dev
              - some-app
              - lambda
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 's3:*'
                  - 'rds-db:connect'
                  - 'rds:*'
                  - 'es:*'
                Resource: '*'
      Path: /
      RoleName: !Join 
        - '-'
        - - some-app
          - dev
          - eu-west-1
          - lambdaRole
      ManagedPolicyArns:
        - !Join 
          - ''
          - - 'arn:'
            - !Ref 'AWS::Partition'
            - ':iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole'

    # ========= The Lambda =========

  AppLambdaFunction:
    Type: 'AWS::Lambda::Function'
    Properties:
      Code:
        S3Bucket: !Ref SomeDeploymentBucket
        S3Key: >-
          tutorial/some-app/dev/1545610972669-2018-12-24T00:22:52.669Z/some-app.zip
      FunctionName: some-lambda
      Handler: app.server
      MemorySize: 1024
      Role: !GetAtt 
        - IamRoleLambdaExecution
        - Arn
      Runtime: nodejs8.10
      Timeout: 6
      VpcConfig:
        SecurityGroupIds:
          - !Ref xxxVPCSecurityGroup
        SubnetIds:
          - !Ref xxxLambdaSubnet
    DependsOn:
      - AppLogGroup
      - IamRoleLambdaExecution

    # ========= VPC =========

  xxxVPC:
    Type: 'AWS::EC2::VPC'
    Properties:
      CidrBlock: 172.31.0.0/16
      InstanceTenancy: default
      EnableDnsSupport: 'true'
      EnableDnsHostnames: 'true'

  xxxVPCSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupName: VPC SG
      GroupDescription: VPC Security Group
      VpcId: !Ref xxxVPC

  xxxLambdaSubnet:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId: !Ref xxxVPC
      CidrBlock: 172.31.32.0/20

    # ========= Elasticsearch =========

  xxxESSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupName: ES SG
      GroupDescription: ES Security group
      VpcId: !Ref xxxVPC
      SecurityGroupIngress:
        - IpProtocol: -1
          FromPort: 0
          ToPort: 65535
          SourceSecurityGroupId: !Ref xxxVPCSecurityGroup

  xxxElasticSearch:
    Type: 'AWS::Elasticsearch::Domain'
    Properties:
      AccessPolicies:
        Version: 2012-10-17
        Statement:
          - Action:
              - 'es:*'
              - 'ec2:*'
              - 's3:*'
            Principal:
              AWS:
                - '*'
            Resource: '*'
            Effect: Allow
      DomainName: es-xxx-domain
      AdvancedOptions:
        rest.action.multi.allow_explicit_index: 'true'
      ElasticsearchVersion: 6.3
      ElasticsearchClusterConfig:
        InstanceCount: 2
        InstanceType: m3.medium.elasticsearch
        DedicatedMasterEnabled: 'false'
      VPCOptions:
        SecurityGroupIds:
          - !Ref xxxESSecurityGroup
        SubnetIds:
          - !Ref xxxLambdaSubnet

The JS code (the version without the signing with creds, but it does not work when signing it either):

var es = require('elasticsearch');
var client = new es.Client({
    host: 'vpc-es-domain-AMAZON.eu-west-1.es.amazonaws.com:80',
    log: 'trace'
});

client.ping({
    requestTimeout: 1000
}, function(error, res, status){
    if(error) {
        console.trace('es cluster error!');
        console.trace(error);
    } else {
        console.log('All is well');
        var response = {
            error: error,
            res: res,
            status: status
        }
        console.log(JSON.stringify(response));
    }
});

Doing this in curl from the EC2 instance in the same VPC gets response from the ES domain without any problems:

curl vpc-es-domain-AMAZON.eu-west-1.es.amazonaws.com:80

I would really appreciate help since I'm already stuck with this.

Upvotes: 6

Views: 7953

Answers (1)

Ankit Gupta
Ankit Gupta

Reputation: 760

Noticed two problems in your setup

  1. You are creating only one subnet in the stack and assigning only one subnet to Lambda. You need to assign multiple subnets to the Lambda
  2. You need to fix access policy for ES. I manually updated to "Don't require signing requst with IAM credential" for my testing in console.

I updated Cloudformation template to create python based lambda handler to query elastic search from the same vpc. It is not complete but if you make sure of the above mentioned issues then it should work.

AWSTemplateFormatVersion: 2010-09-09
Description: The AWS CloudFormation tutorial
Resources:

  SomeDeploymentBucket:
    Type: 'AWS::S3::Bucket'

  AppLogGroup:
    Type: 'AWS::Logs::LogGroup'
    Properties:
      LogGroupName: /aws/lambda/some-lambda

    # ========= The Lambda Execution Role =========

  IamRoleLambdaExecution:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Policies:
        - PolicyName: !Join
            - '-'
            - - dev
              - some-app
              - lambda
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 's3:*'
                  - 'rds-db:connect'
                  - 'rds:*'
                  - 'es:*'
                Resource: '*'


      Path: /
      RoleName: !Join
        - '-'
        - - some-app
          - dev
          - eu-west-1
          - lambdaRole
      ManagedPolicyArns:
        - !Join
          - ''
          - - 'arn:'
            - !Ref 'AWS::Partition'
            - ':iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole'

    # ========= The Lambda =========

  AppLambdaFunction:
    Type: AWS::Lambda::Function
    DependsOn:
      - AppLogGroup
      - IamRoleLambdaExecution
    Properties:
      FunctionName: some-lambda
      Handler: index.lambda_handler
      Runtime: python2.7
      Timeout: 60
      MemorySize: 1024
      Role: !GetAtt
        - IamRoleLambdaExecution
        - Arn
      VpcConfig:
        SecurityGroupIds:
          - !Ref xxxVPCSecurityGroup
        SubnetIds:
          - !Ref xxxLambdaSubnet1
          - !Ref xxxLambdaSubnet2

      Code:
        ZipFile: !Sub |
          from __future__ import print_function
          import boto3
          iam = boto3.client('iam')

          def lambda_handler(event, context):
            print('called lambda_handler')

    # ========= VPC =========

  xxxVPC:
    Type: 'AWS::EC2::VPC'
    Properties:
      CidrBlock: 172.31.0.0/16
      InstanceTenancy: default
      EnableDnsSupport: 'true'
      EnableDnsHostnames: 'true'

  xxxVPCSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupName: VPC SG
      GroupDescription: VPC Security Group
      VpcId: !Ref xxxVPC

  xxxLambdaSubnet1:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId: !Ref xxxVPC
      CidrBlock: 172.31.32.0/20
  xxxLambdaSubnet2:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId: !Ref xxxVPC
      CidrBlock: 172.31.16.0/20


    # ========= Elasticsearch =========

  xxxESSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupName: ES SG
      GroupDescription: ES Security group
      VpcId: !Ref xxxVPC
      SecurityGroupIngress:
        - IpProtocol: -1
          FromPort: 0
          ToPort: 65535
          SourceSecurityGroupId: !Ref xxxVPCSecurityGroup

  xxxElasticSearch:
    Type: 'AWS::Elasticsearch::Domain'
    Properties:
      AccessPolicies:
        Version: 2012-10-17
        Statement:
          - Action:
              - 'es:*'
              - 'ec2:*'
              - 's3:*'
            Principal:
              AWS:
                - '*'
            Resource: '*'
            Effect: Allow
      DomainName: es-xxx-domain
      EBSOptions:
        EBSEnabled: true
        Iops: 0
        VolumeSize: 20
        VolumeType: "gp2"
      AdvancedOptions:
        rest.action.multi.allow_explicit_index: 'true'

      ElasticsearchClusterConfig:
        InstanceCount: 1
        InstanceType: m4.large.elasticsearch
        DedicatedMasterEnabled: 'false'
      VPCOptions:
        SecurityGroupIds:
          - !Ref xxxESSecurityGroup
        SubnetIds:
          - !Ref xxxLambdaSubnet1

Updated Lambda function handler code

  import urllib2

def lambda_handler(event, context):
    print('called lambda_handler')
    data = ''
    url = 'https://vpc-es-xxx-domain-fixthis.es.amazonaws.com'
    req = urllib2.Request(url, data, {'Content-Type': 'application/json'})
    f = urllib2.urlopen(req)
    for x in f:
        print(x)
    f.close()

Upvotes: 3

Related Questions