CODEWITHSUNDEEP
amazon-web-servicesamazon-iamprometheusaws-fargate
Aaron M
Aaron M

Reputation: 391

Fargate task AWSSecurityTokenService Access Denied

I am trying to get Prometheus' CloudWatch Exporter running as a Fargate task. I am building a custom image with the config file baked in based on the prom/cloudwatch-exporter image.

When the container comes up I am seeing the following error in the logs:

com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: Access denied (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: REQUEST-ID)

The call that produces that error appears to be this:

at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.listMetrics(AmazonCloudWatchClient.java:684)

Both the Task Execution Role and the Task role have the following policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:GetMetricStatistics",
                "cloudwatch:ListMetrics"
            ],
            "Resource": "*"
        }
    ]
}

Here is the container definition:

{
      "dnsSearchDomains": null,
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "LOG-GROUP",
          "awslogs-region": "REGION",
          "awslogs-stream-prefix": "LOG-PREFIX"
        }
      },
      "entryPoint": null,
      "portMappings": [
        {
          "hostPort": 9106,
          "protocol": "tcp",
          "containerPort": 9106
        }
      ],
      "command": null,
      "linuxParameters": null,
      "cpu": 0,
      "environment": [],
      "resourceRequirements": null,
      "ulimits": null,
      "dnsServers": null,
      "mountPoints": [],
      "workingDirectory": null,
      "secrets": null,
      "dockerSecurityOptions": null,
      "memory": null,
      "memoryReservation": null,
      "volumesFrom": [],
      "image": "ACCOUNTID.dkr.ecr.REGION.amazonaws.com/mycustomimage:latest",
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": null,
      "essential": true,
      "links": null,
      "hostname": null,
      "extraHosts": null,
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": null,
      "dockerLabels": null,
      "systemControls": null,
      "privileged": null,
      "name": "container-name"
    }

Why is the container not authenticating based on the IAM policies? Every other policy in the setup appears to be working as expected. The cluster can pull the custom image from the ECR repo, logs are being written, etc.

Upvotes: 2

Views: 888

Answers (1)

Aaron M
Aaron M

Reputation: 391

I figured it out. CloudWatch Exporter lets you pass the IAM role arn via the config attribute role_arn. If this value is set, then the app uses STSAssumeRoleSessionCredentialsProvider to establish credentials. This is apparently not supported in Fargate (this method works in EC2-based ECS containers). If you leave off role_arn then the app creates a new client with default settings, which uses the DefaultAWSCredentialsProviderChain class and this works like a charm.

Upvotes: 1

Related Questions