CODEWITHSUNDEEP
python-2.7amazon-web-serviceselasticsearchboto3
Alec
Alec

Reputation: 2752

Making a signed HTTP request to AWS Elasticsearch in Python

I'm trying to make a simple Python Lambda that makes snapshots of our Elasticsearch database. This is done through Elasticsearch's REST API using simple HTTP requests.

However, for AWS, I have to sign these requests. I have a feeling it can be achieved through boto3's low-level clients probably with generate_presigned_url, but I cannot for the life of me figure out how to invoke this function correctly. For example, what are the valid ClientMethods? I've tried ESHttpGet but to no avail.

Can anyone point me in the right direction?

Upvotes: 15

Views: 15802

Answers (5)

Alex Rudd
Alex Rudd

Reputation: 326

Edit: Apparently this workaround has been broken by Elastic.

I struggled for a while to do a similar thing. Currently the boto3 library doesn't support making signed es requests, though since I raised an issue with them it's become a feature request.

Here's what I've done in the meantime using DavidMuller's library mentioned above and boto3 to get my STS session credentials:

import boto3
from aws_requests_auth.aws_auth import AWSRequestsAuth
from elasticsearch import Elasticsearch, RequestsHttpConnection

session = boto3.session.Session()
credentials = session.get_credentials().get_frozen_credentials()

es_host = 'search-my-es-domain.eu-west-1.es.amazonaws.com'
awsauth = AWSRequestsAuth(
    aws_access_key=credentials.access_key,
    aws_secret_access_key=credentials.secret_key,
    aws_token=credentials.token,
    aws_host=es_host,
    aws_region=session.region_name,
    aws_service='es'
)

# use the requests connection_class and pass in our custom auth class
es = Elasticsearch(
    hosts=[{'host': es_host, 'port': 443}],
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)

print(es.info())

Hope this saves somebody some time.

Upvotes: 19

Pruthvi Raj
Pruthvi Raj

Reputation: 624

why not just use requests?

import requests
headers = {'Content-Type': 'application/json',}
data = '{"director": "Burton, Tim", "genre": ["Comedy","Sci-Fi","R-rated"],"profit" : 98 , "year": 1996, "actor": ["Jack Nicholson","PierceBrosnan","Sarah Jessica Parker"], "title": "Mars Attacks!"}'
response = requests.post('https://search-your-awsendpoint.us-west-2.es.amazonaws.com/yourindex/_yourdoc/', headers=headers, data=data)

this worked for me

Upvotes: 0

b.b3rn4rd
b.b3rn4rd

Reputation: 8830

While other answers are perfectly fine, I wanted to eliminate the use of external packages. Obviously, botocore itself has all the required functionality to sign requests it was just a matter of looking at the source code. This is what I ended up with for sending AWS API requests directly (things are hardcoded for the demonstration purposes):

      import boto3
      import botocore.credentials
      from botocore.awsrequest import AWSRequest
      from botocore.endpoint import URLLib3Session
      from botocore.auth import SigV4Auth

      params = '{"name": "hello"}'
      headers = {
        'Host': 'ram.ap-southeast-2.amazonaws.com',
      }
      request = AWSRequest(method="POST", url="https://ram.ap-southeast-2.amazonaws.com/createresourceshare", data=params, headers=headers)
      SigV4Auth(boto3.Session().get_credentials(), "ram", "ap-southeast-2").add_auth(request)    


      session = URLLib3Session()
      r = session.send(request.prepare())

Upvotes: 13

mixja
mixja

Reputation: 7467

I recently published requests-aws-sign, which provides AWS V4 request signing for the Python requests library.

If you look at this code you will see how you can use Botocore to generate the V4 request signing.

Upvotes: 4

garnaat
garnaat

Reputation: 45846

There are several Python extensions to the requests library that will perform the SigV4 signing for you. I have used this one and it works well.

Upvotes: 10

Related Questions