CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaamazon-elasticsearch
Keith
Keith

Reputation: 590

Permissions to access ElasticSearch from Lambda?

I'm trying to use Elasticsearch for data storage for a Lambda function connected to Alexa Skills Kit. The Lambda works alright without Elasticsearch but ES provides much-needed fuzzy matching.

The only way I've been able to access it from Lambda is by enabling Elasticsearch global access but that's a really bad idea. I've also been able to access from my computer via open access policy or IP address policy. Is there a way to do read-only access via Lambda and read-write via IP?

On IAM I granted my Lambda role AmazonESReadOnlyAccess. On the ES side I tried this but it only worked for IP address:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::NUMBER:root",
          "arn:aws:iam::NUMBER:role/lambda_basic_execution"
        ]
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:NUMBER:domain/NAME/*"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:NUMBER:domain/NAME/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": "MY IP"
        }
      }
    }
  ]
}

This forum post asks the same question but went unanswered.

Upvotes: 29

Views: 34350

Answers (5)

Yurii Holskyi
Yurii Holskyi

Reputation: 918

Now it's possible from your code with elasticsearch.js. Before you try it, you must install http-aws-es module.

const AWS = require('aws-sdk');
const httpAwsEs = require('http-aws-es');
const elasticsearch = require('elasticsearch');

const client = new elasticsearch.Client({
    host: 'YOUR_ES_HOST',
    connectionClass: httpAwsEs,
    amazonES: {
        region: 'YOUR_ES_REGION',
        credentials: new AWS.EnvironmentCredentials('AWS')
    }
});

// client.search({...})

Of course, before using it, configure access to elasticsearch domain: enter image description here

Upvotes: 15

Shef
Shef

Reputation: 57

You need to go to the access policy of Lambda and provide the AWS ARN to connect

http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-aws-integrations.html#es-aws-integrations-s3-lambda-es-authorizations

Upvotes: 4

Brooks
Brooks

Reputation: 7380

For external (outside AWS) access to your Elasticsearch cluster, you want to create the cluster with an IP-based access policy. Something like the below:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "<<IP/CIDR>>"
          ]
        }
      },
      "Resource": "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/*"
    }
  ]
}

For your Lambda function, create the role that the Lambda function will assume with the below policy snippet. 

{
  "Sid": "",
  "Effect": "Allow",
  "Action": [
    "es:DescribeElasticsearchDomain",
    "es:DescribeElasticsearchDomains",
    "es:DescribeElasticsearchDomainConfig",
    "es:ESHttpPost",
    "es:ESHttpPut"
  ],
  "Resource": [
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>",
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/*"
  ]
},
{
  "Sid": "",
  "Effect": "Allow",
  "Action": [
    "es:ESHttpGet"
  ],
  "Resource": [
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/_all/_settings",
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/_cluster/stats",
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/<<INDEX>>*/_mapping/<<TYPE>>",
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/_nodes",
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/_nodes/stats",
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/_nodes/*/stats",
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/_stats",
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/<<INDEX>>*/_stats"
  ]
}

I think you could more easily condense the above two policy statements into the following:

{
  "Sid": "",
  "Effect": "Allow",
  "Action": [
    "es:DescribeElasticsearchDomain",
    "es:DescribeElasticsearchDomains",
    "es:DescribeElasticsearchDomainConfig",
    "es:ESHttpPost",
    "es:ESHttpGet",
    "es:ESHttpPut"
  ],
  "Resource": [
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>",
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/*"
  ]
}

I managed to piece the above together from the following sources:

https://aws.amazon.com/blogs/security/how-to-control-access-to-your-amazon-elasticsearch-service-domain/

How to access Kibana from Amazon elasticsearch service?

https://forums.aws.amazon.com/thread.jspa?threadID=217149

Upvotes: 5

Raheel Kazi
Raheel Kazi

Reputation: 49

AWS Lambda runs on public EC2 instances. So simply adding a whitelist of IP addresses to the Elasticsearch access policy will not work. One way to do this will be to give the Lambda execution role appropriate permissions to the Elasticsearch domain. Make sure that the Lambda Execution role has permissions to the ES domain and the ES domain access policy has a statement which allows this Lambda Role ARN to do the appropriate actions. Once this is done all you would have to do is sign your request via SigV4 while accessing the ES endpoint

Hope that helps!

Upvotes: 1

garnaat
garnaat

Reputation: 45846

The only way I know of to do this is to use a resource-based policy or an IAM-based policy on your ES domain. This would restrict access to a particular IAM user or role. However, to make this work you also need to sign your requests to ES using SigV4.

There are libraries that will do this signing for you, for example this one extends the popular Python requests library to sign ElasticSearch requests via SigV4. I believe similar libraries exist for other languages.

Upvotes: 16

Related Questions