CODEWITHSUNDEEP
amazon-web-serviceselasticsearchhibernate-searchamazon-iamaws-codebuild
bertie
bertie

Reputation: 503

AWS Elastisearch Access Policy for CodeBuild Integration Tests with Hibernate Search using a ElasticSearch for indexes storage

I want to launch a CodeBuild project to run my integration tests. My application use AWS ElasticSearch Service as Hibernate Search index storage.

I have added a policy to my ES Domain which allows private ec2 instances to access ES through a NAT Gateway. Unfortunally I can't figured out the correct policy to allow CodeBuild access ES. When I run CodeBuild project I get a 403 error when Hibernate try to check an index existence.

Caused by: org.hibernate.search.exception.SearchException:   HSEARCH400007: Elasticsearch request failed.
Request:
Operation: IndicesExists
URI:com.mycompany.myproject.model.tenant
Data:
null
Response:
=========
Status: 403
Error message: 403 Forbidden
Cluster name: null
Cluster status: null

I tried to configured ES Access Policy to allow open access to the domain, then tests runs ok ("AWS": "*").

This is the ES Access Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
          "AWS": "arn:aws:iam::AWS_ACCOUNT_ID:role/CodeBuildRole-XXXXXXXX"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-west-1:AWS_ACOUNT_ID:domain/elastic-search-domain/*"      
  },
  {
    "Effect": "Allow",
    "Principal": {
       "AWS": "*"
    },
    "Action": "es:*",
    "Resource": "arn:aws:es:eu-west-1:AWS_ACCOUNT_ID:domain/elastic-search-domain/*",
    "Condition": {
      "IpAddress": {
        "aws:SourceIp": "NAT_GW_IP"
      }
     }
   }
  ]
 }

As principal I've also tried the following:

"arn:aws:sts::AWS_ACCOUNT_ID:assumed-role/CodeBuildRole-XXXXXXXXX/*" "arn:aws:iam::AWS_ACCOUNT_ID:role/CodeBuildRole-XXXXXXXXX" "arn:aws:iam::AWS_ACCOUNT_ID:root" "arn:aws:iam::AWS_ACCOUNT_ID:user/MI_USER_ADMIN"

Any help will be very appreciated.

Thanks

Upvotes: 3

Views: 489

Answers (2)

bertie
bertie

Reputation: 503

I would like to extend the VME answer to be more precise.

To access ElasticSearch using a role, the request must certainly be signed.

This solution is generally correct, but on my particular case this is not suitable since the requests to AWS ES are generated by Hibernate Search ElasticSearch. (Might we find another solution using AOP?)

I finally figured out a workaround for this problem. On CodeBuild build spec I added the following steps:

  • Configure AWS CLI using a user with a policy that allows him to read and update ES domain.
  • Read and store the current ES Domain Access Policy
  • I get the CodeBuild ec2 IP
  • Update ES Domain policy access to allow access from CodeBuild IP
  • Wait until the changes applies (15 mins aprox)
  • Run the test
  • Restore the previous configuration

I don't like this solution very much because the Domain Policy updates takes too long. This step is part of a CodePipeline for Continuous Integration, and executions should not take more than 15 or 20 minutes.

Any ideas on how to improve this?

Upvotes: 2

veuncent
veuncent

Reputation: 1723

Possibly you need to sign your ES requests.

I am not familiar with CodeBuild, but generally the rule is: when using IAM roles to access Elasticsearch, your requests need to be signed with that IAM role.

E.g. For python you would use a tool like this: https://github.com/DavidMuller/aws-requests-auth

More info: http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html

Upvotes: 1

Related Questions