CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-lambda
zephel
zephel

Reputation: 185

How to trigger Lambda only one time for multipart upload S3 triggers?

I have a AWS Lambda function that:

  1. reacts to an S3 ObjectCreated trigger
  2. ssh into an ec2 instance and
  3. runs a python script

This python script will then run EMR to process all these S3 part-files that were just created. (Yes, the files must be processed jointly.)

However, the part-files (file_part_0000, file_part_0001, file_part_0002, etc.) are created as multipart uploads.

Lambda Event: {..., "eventName": "ObjectCreated:CompleteMultipartUpload", ...}

It's my understanding that this trigger would force Lambda to ssh and run the python script as many time as there are part files. I only want Lambda to ssh and run the script once.

Is there any way to have the Lambda be triggered once for each "series" of multipart uploads?

Upvotes: 3

Views: 5590

Answers (1)

mootmoot
mootmoot

Reputation: 13176

As mentioned in Configuring Amazon S3 Event Notifications

You will not receive event notifications from failed operations.

And there is a reason that the event is call s3:ObjectCreated:CompleteMultipartUpload, the event should trigger only if your multipart are upload completely.

On the other hand, perhaps you should make S3 event trigger to send info to SQS. So you can make your python program can watching SQS queue and process the file. With SQS, you don't need to make passwordless setup or store private key in lambda to ssh the server.

Upvotes: 1

Related Questions