CODEWITHSUNDEEP
amazon-rdsamazon-cloudwatchaws-glueamazon-cloudwatchlogs
Shaun
Shaun

Reputation: 535

AWS Glue Test Connection Logs

I've set up an AWS Glue connection to an RDS database (in the same account and region). When doing the test connection I get the following error

rds-prod-snapshot test connection failed. For more information see the logs

Following the link to CloudWatch I get the error

There was an error getting log events.
The specified log stream does not exist.

The role has IAM permissions for CloudWatch logs

I followed the troubleshooting doc to get this far > https://aws.amazon.com/premiumsupport/knowledge-center/glue-test-connection-failed/ (in fact I got most of these resolved when setting up the connection in the first place and resolved)

Upvotes: 2

Views: 1414

Answers (2)

Felipe
Felipe

Reputation: 7573

Just to add another my 2 cents to this question. Although you can attach the AWSGlueServiceRole policy to the role, it is also possible to define a specific policy just for the log group. IMHO it is better to define it so that you can be specific on the resources that each action is using.

  statement {
    sid    = "CloudWatchGlueJobLogs"
    effect = "Allow"
    actions = [
      "logs:GetLogEvents",
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]
    resources = [
      "arn:aws:logs:*:*:log-group:/aws-glue/jobs/output:*",
      "arn:aws:logs:*:*:log-group:/aws-glue/jobs/error:*",
      "arn:aws:logs:*:*:log-group:/aws-glue/jobs/logs-v2:*",

      # THERE 2 log-groups ARE RELEVANT TO YOUR ERROR
      # I ADDED OTHER POLICIES JUST FOR COMPLETENESS

      "arn:aws:logs:*:*:log-group:/aws-glue/testconnection/output/your_glue_connection:*",
      "arn:aws:logs:*:*:log-group:/aws-glue/testconnection/error/your_glue_connection:*"
    ]
  }
  statement {
    sid    = "GlueJobRead"
    effect = "Allow"
    actions = [
      "glue:GetDatabase*",
      "glue:GetTable",
      "glue:GetPartition*",
      "glue:BatchGetPartition",
      "glue:GetConnection"
    ]
    resources = [
      "arn:aws:glue:*:*:catalog",
      "arn:aws:glue:*:*:database/default",
      "arn:aws:glue:*:*:database/your_ddatabase",
      "arn:aws:glue:*:*:table/your_database/your_table",
      "arn:aws:glue:*:*:table/default/table_name",
      "arn:aws:glue:*:*:connection/your_glue_connection"
    ]
  }
  statement {
    sid       = "CloudwatchMetrics"
    effect    = "Allow"
    actions   = ["cloudwatch:PutMetricData"]
    resources = ["*"]
    condition {
      test     = "StringEquals"
      values   = ["Glue"]
      variable = "cloudwatch:namespace"
    }
  }
  statement {
    sid    = "GlueJobConnectionToRDS"
    effect = "Allow"
    actions = [
      "ec2:DescribeSubnets",
      "ec2:DescribeSecurityGroups",
      "ec2:DescribeVpcEndpoints",
      "ec2:DescribeRouteTables",
      "ec2:CreateNetworkInterface",
      "ec2:DeleteNetworkInterface",
      "ec2:DescribeNetworkInterfaces",
      "ec2:DescribeVpcAttribute",
    ]
    resources = ["*"]
  }

Upvotes: 0

Shaun
Shaun

Reputation: 535

Turns out AWSGlueServiceRole policy was not attached to the role

Upvotes: 3

Related Questions