CODEWITHSUNDEEP
databaseamazon-web-servicesaws-cloudformationamazon-iamamazon-athena
Michael Barrowman
Michael Barrowman

Reputation: 1201

AWS Glue missing permissions

I am trying to use cloudformation to create a table in AWS Glue/Athena. However, I am being hit by the following error in the Cloudformation Events panel:

Cross account access is not supported for account that hasn't imported Athena catalog to Glue. Please refer to documentation: https://docs.aws.amazon.com/athena/latest/ug/glue-upgrade.html (Service: AWSGlue; Status Code: 400; Error Code: AccessDeniedException; Request ID: <REMOVED>; Proxy: null)

(I have removed the Request-ID)

I followed the link, and it is information on how to upgrade to using the Glue Data Catalog within Athena. I do not think this is the actual problem I am facing. However, I ensured that the IAM permissions were correct as per the guide, but the final step says to "choose Upgrade in the Athena console", which is not present. I can only assume that means this has already been done as I have already been using Glue with Athena

Within AWS, I have a main account (which I use within the console with very broad permissions) and a User account (which I manage and control the IAM permissions, and mostly use for programmatic access via AWS-CLI, boto3, etc...).

My main (Console) account created everything that is currently in Glue/Athena. However, my local (User) account has access to everything in Glue (e.g. {"Effect":"Allow", "Action":["glue:*","athena:*","s3:*"], "Resource":"*"}). I tested this in a few ways

From my Local account, I can list the Glue Catalog:

>aws athena list-data-catalogs
{
    "DataCatalogsSummary": [
        {
            "CatalogName": "AwsDataCatalog",
            "Type": "GLUE"
        }
    ]
}

I am also able to list all the tables within the database I am trying to access using:

aws glue get-tables --database-name <DBNAME>

And I was able to create a dummy table using the cli:

aws glue create-table --database-name [[DBNAME]] --table-input "Name:TestTable"

(again, I have removed the database name):

Googling for the error above leads to instructions on how to set up Cross Account Access, which I have done by ensuring the following is in my Glue Catalog Settings Permissions ( censored appropriately):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<ACCOUNTID>:user/<Local Username>"
                ]
            },
            "Action": "glue:*",
            "Resource": [
                "arn:aws:glue:us-east-1:<ACCOUNTID>:catalog",
                "arn:aws:glue:us-east-1:<ACCOUNTID>:database/*",
                "arn:aws:glue:us-east-1:<ACCOUNTID>:table/*"
            ]
        }
    ]
}

Upvotes: 0

Views: 1534

Answers (2)

Emerson Chua
Emerson Chua

Reputation: 1

I encountered this error when using boto3. Using the AWS Account ID worked for me. This is what my function ended up looking like.

def create_catalog_database():
    # Create a Glue client
    session = boto3.session.Session()
    client = session.client(
        service_name='glue',
        region_name=os.getenv("AWS_REGION")
    )

    try:
        response = client.create_database(
            CatalogId=ACCOUNT_ID,
            DatabaseInput={
                "Name": CATALOG_DB
            }
        )
    except client.exceptions.AlreadyExistsException:
        logging.info(f"Database {DEST_BUCKET_ARN} already exits!")
        response = None
    
    return response

Upvotes: -1

Michael Barrowman
Michael Barrowman

Reputation: 1201

After contacting AWS support, I found the solution, so I'm posting it here in case anyone else encounters this problem in the future. The CatalogId is the AccountID, and not the name of the Catalog as seen in the Athena Console. The solution was to replace the CatalogId in the template with !Sub '${AccountId}'. For example:

GlueTable:
  Type: AWS::Glue::Table
    Properties:
      CatalogId: !Sub '${AWS::AccountId}'
      DatabaseName: !Sub 'db_{BucketName}'
      TableInput:
        Name: 'tbl_{LocalName}'
        Description: !Sub 'Glue Table for {LocalName}'
....

Upvotes: 0

Related Questions