CODEWITHSUNDEEP
amazon-web-servicesamazon-s3amazon-cloudwatchevent-driven
Edge7
Edge7

Reputation: 681

AWS event driven approach - Cloud Watch vs S3 event notification

I am building an event-driven system, which starts as soon as a new file lands S3. I am evaluating different ways of achieving that and using Cloud Watch Rule + API Trail is an option. This is the Cloud Watch Event pattern as it is:

    {
  "source": [
    "aws.s3"
  ],
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "detail": {
    "eventSource": [
      "s3.amazonaws.com"
    ],
    "eventName": [
      "PutObject"
    ],
    "requestParameters": {
      "bucketName": [
        "mysupertest88"
      ]
    }
  }
}

Like that, it triggers the rule for every file landing into the bucket but trying to filter by key and wildcard does not work:

"requestParameters": {
      "bucketName": [
        "mysupertest88"
      ],
      "key": ["myprefix/mysecondprefix/*"]
    }

It works just if I specify a key with matches without a wildcard, I think because the symbol '*' is a valid char in S3 objects. An alternative is to filter directly at Trail level: API Trail

but I do not see that as a nice option, as API Trail is often out of developer's control. An additional alternative is to use content-filtering: (nice new feature, but you have to create the rule through EventBridge)

    {
  "source": [
    "aws.s3"
  ],
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "detail": {
    "eventSource": [
      "s3.amazonaws.com"
    ],
    "eventName": [
      "PutObject"
    ],
    "requestParameters": {
      "bucketName": [
        "mysupertest88"
      ],
      "key": [
        {
          "prefix": "a/c"
        }
      ]
    }
  }
}

Last S3 event notification is the old way to accomplish this? What is your experience with that? Any pro and cons that are not easy to catch without experience?

Upvotes: 1

Views: 743

Answers (2)

aerioeus
aerioeus

Reputation: 1410

enter image description hereyes, s3 notifications are much faster:

https://eoins.medium.com/comparing-two-ways-to-trigger-lambda-from-s3-b5da8cfe1aee

Upvotes: 0

Marcin
Marcin

Reputation: 238051

Since your objective is to start an action "as soon as a new file lands S3", CT may not satisfy your requirement. This is because it can take up 15 minutes for delivery of the API events. From AWS faq:

Typically, CloudTrail delivers an event within 15 minutes of the API call.

In contrast, S3 events should be faster. From AWS docs:

Amazon S3 event notifications are designed to be delivered at least once. Typically, event notifications are delivered in seconds but can sometimes take a minute or longer.

Upvotes: 3

Related Questions