CODEWITHSUNDEEP
amazon-web-servicespostmanaws-glue
alexopoulos7
alexopoulos7

Reputation: 912

Glue aws connect with Web Api

I have setup aws glue crawlers and have already databases with tables populated to my Glue Data Catalog. I would like to access information on Data Catalog using Web API.

For example I would like to GetDatabases.

When I am using python boto3 library I get the list of all databases.

import boto3
glue = boto3.client('glue',region_name='us-west-2') 
glue.get_databases()

The same when using aws-sdk js library

var AWS = require('aws-sdk');
AWS.config.update({region: 'us-west-2'});
var glue = new AWS.Glue();

glue.getDatabases((err,data)=>{
    if(err){
        console.log(err);
    }
    console.log(data);
});

What I am trying to do is to call the web api from Postman. Postman creates the AWS Signature (under authorization tab) using the AWS credentials.

When I use postman for describing an ec2 instance it works fine. However, I get an AccessDeniedException when trying to do a 

POST https://glue.us-west-2.amazonaws.com/
Headers: X-Amz-Target: AWSGlue.GetDatabases
Authorization:....
X-Amz-Date:...
Content-Type:multipart/form-data

Am I missing some header? How can I invoke web api and get a list of databases?

Thanks for your help.

Upvotes: 1

Views: 2054

Answers (1)

alexopoulos7
alexopoulos7

Reputation: 912

After a lot of trial and error, I have found out that the following Headers are important

POST / HTTP/1.1
Host: glue.us-west-2.amazonaws.com
X-Amz-Target: AWSGlue.GetDatabases
Content-Type: application/x-amz-json-1.1
X-Amz-Date: 20180425T052803Z
Authorization: AWS4-HMAC-SHA256 Credential=AWS_KEY/20180425/us-west-2/glue/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-target, Signature=f6dfb78d7ee837dc57ce6e13d9fd6ec0631d0c5546f0142ce997bc7e9203c7b7

Also, something important is to pass an empty body ({}) if you do not want to specify any of the parameters.

 curl --location --request POST 'https://glue.us-west-2.amazonaws.com/databases?CatalogId=XXXXXXXXXXXX' \
            --header 'X-Amz-Target: AWSGlue.GetDatabases' \
            --header 'Content-Type: application/x-amz-json-1.1' \
            --header 'X-Amz-Content-Sha256: mplamplamplampla' \
            --header 'X-Amz-Date: 20230218T084937Z' \
            --header 'Authorization: AWS4-HMAC-SHA256 Credential=AWS_KEY/20230218/us-west-2/glue/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date;x-amz-target, Signature=123123123123123123123123123' \
            --data-raw '{}'

Details

  1. In the Params Section add your CatalogId value
  2. In the Auth Section Select as Type: AWS Signature and fill in your Access Key, Secret Key and Region
  3. In the Headers Section set up X-Amz-Target, Content-Type and X-Amz-Date as above and in the
  4. In the Body Section select raw and put emptu curly braces ({}) in the body

Upvotes: 3

Related Questions