CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-s3amazon-ec2aws-lambda
Dave
Dave

Reputation: 349

AWS S3 Notification Consolidation

My goal is to develop an application that can check my AWS S3 bucket for five files and once achieved, kick off the Lambda function that loads these files into my database. Loading files from an S3 event is working well. The issue I face is that, even when you upload all files at the same time, each file generates a single event. My loading function gets kicked off five separate times.

I have some ideas on how to solve this issue. One approach is to consolidate the event notifications into one event. Is this possible? This approach would work because then I could have the process break if the client only uploads 4 files (and subsequently send an email saying to upload 5 files). Otherwise, in the event of 5 files, I can then kick off the Lambda to process everything in the bucket (if # objects in bucket = 5, of course! Which can be checked very easily.) This way, the Lambda function only runs one time.

In summary, I want the event notifications to be consolidated so I can run the Lambda function once instead of five times. Please advise if this is possible, and through which channel of AWS product I can find my solution in. Thank you.

Upvotes: 1

Views: 319

Answers (1)

John Rotenstein
John Rotenstein

Reputation: 270184

There is no such concept as "event consolidation".

You will need to:

  • Trigger an AWS Lambda function on every upload
  • That Lambda function should first check that all 5 files have been upload
  • If so, it can process the files. If not, it should exit.

The hard part is sending a notification if they "only upload 4 files", because this would require waiting sufficient time to allow them to upload the remaining file(s) before sending a notification. This would suggest using an AWS Step Function with a Wait so that it can check the files after a given period. The Step Function could be triggered on the first file upload (that is, when the above Lambda function sees only one file).

Upvotes: 2

Related Questions