CODEWITHSUNDEEP
amazon-s3terraformamazon-ecsamazon-cloudwatch
Aaron Zhong
Aaron Zhong

Reputation: 1022

How to use input transformer for ECS Fargate launch type with Terraform CloudWatch event trigger

I'm using terraform to create a CloudWatch Event Trigger with a ECS Fargate launch type which the event source is S3. When I use the input_transformer field to pass in the bucket and key into the ECS task, my event rule results in a failed invocation.

This is the aws_cloudwatch_event_rule:

resource "aws_cloudwatch_event_rule" "event_rule" {
  name          = "dev-gnss-source-put-rule-tf"
  description   = "Capture S3 events on uploads bucket"
  event_pattern = <<PATTERN
{
  "source": [
    "aws.s3"
  ],
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "detail": {
    "eventSource": [
      "s3.amazonaws.com"
    ],
    "eventName": [
      "PutObject"
    ],
    "requestParameters": {
      "bucketName": [
        "example-bucket-name"
      ]
    }
  }
}
PATTERN
}

This is the aws_cloudwatch_event_target:

resource "aws_cloudwatch_event_target" "event_target" {
  target_id = "dev-gnss-upload-event-target-tf"
  arn       = "example-cluster-arn"
  rule      = aws_cloudwatch_event_rule.event_rule.name
  role_arn  = aws_iam_role.uploads_events.arn
  ecs_target {
    launch_type = "FARGATE"
    task_count  = 1 # Launch one container / event
    task_definition_arn = "example-task-definition-arn"
    network_configuration {
      subnets         = ["example-subnet"]
      security_groups = []
    }
  }

  input_transformer {
    input_paths = {
      s3_bucket = "$.detail.requestParameters.bucketName"
      s3_key    = "$.detail.requestParameters.key"
    }

    input_template = <<TEMPLATE
{
  "containerOverrides": [
    {
      "name": "myproject-task",
      "environment": [
        { "name": "S3_BUCKET", "value": <s3_bucket> },
        { "name": "S3_KEY", "value": <s3_key> }
      ]
    }
  ]
}
TEMPLATE
  }
}

If I remove the input_transformer section, it will work fine, but I need to pass in the s3 bucket and key to process the particular file.

My rationale for doing this is to remove the need for an intermediary Lambda and was guided by this Medium post: https://medium.com/@bowbaq/trigger-an-ecs-job-when-an-s3-upload-completes-3559c44c37d1

Any advice is appreciated.

Upvotes: 7

Views: 3419

Answers (1)

Aaron Zhong
Aaron Zhong

Reputation: 1022

After hours of going in circles, I found an answer!

So the first step is to check what the cause of the failed invocation is. You can do this by checking CloudTrail logs by navigating to Cloud Trail > Event history > Search by Event name and type RunTask in the search box. You should see a series of events from the event source ecs.amazonaws.com. Find one that relates to your the Failed Invocation you experienced.

When you click into the event, you can see under the Event record section an errorMessage. In my case, it was the following:

   "errorCode": "InvalidParameterException",
   "errorMessage": "Override for container named myproject-task is not a container in the TaskDefinition.",

This may be different for you. For me, it was because my containerOverride name was incorrect. This field refers to: The name of the container that receives the override. This parameter is required if any override is specified. ref: https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerOverride.html

Correcting this field fixed my issue.

Upvotes: 8

Related Questions