CODEWITHSUNDEEP
amazon-web-servicesaws-lambda
L Xandor
L Xandor

Reputation: 1841

Check for multiple s3 events to trigger lambda when all are met?

I have a step function that runs a series of lambda functions. I would like to set up a lambda that will trigger this each day when ALL of the 3 input files have been updated.

I know you can trigger off of s3 events, but how can I have the requirement that all three have to have been updated? That is, I don't want the trigger to fire on any of the three events ('or' condition), I want it to trigger when all three have been updated ('and' condition).

How would I approach setting up such a trigger event?

Thanks

Upvotes: 3

Views: 2167

Answers (3)

L Xandor
L Xandor

Reputation: 1841

Here is the solution I ended using. I'm using a single lambda with a trigger for each file. For each trigger, the lambda adds the name of the file that triggered it to a file stored on s3. It then checks if the file has all the required file names to continue. If it does, it kicks off the stepfunction and clears out the list. I am capturing the name of the file that triggers the lambda from the event object that is passed in by default via lambda_function(event,context).

So on a given day, the same lambda gets triggered multiple times, recording the filename each time until it has collected them all.

Upvotes: 1

CK__
CK__

Reputation: 1311

As if now, there is no direct way to tigger Lambda on the basis of such conditions. You need to write your logic in a separate lambda which will trigger your step function.

You can refer here about how to trigger a Step Function from Lambda.

Now, coming to the point about logic in your Lambda which will check for your files. Here, I believe you know all those three files or at-least you know the file prefix so when ever any file is uploaded to S3 you can check in Lambda if three files are there or not. You can trigger the state machine directly from Lambda if all three files are found else you can stop there itself. Once the Step Machines execution is complete either you can clean up those three files or you can rename it to something else or you can move those files to a different "folder" so that next time when you upload any of those three files your lambda does not triggers state machine.

The other logic I could think of is, uploading all these files together as zip. Once the zip is uploaded to S3, your lambda gets notified, processes that zip and extracts all three files, does some processing and triggers the state machine with processed data.

Upvotes: 1

John Rotenstein
John Rotenstein

Reputation: 270224

There is no way to define a trigger that says "only run after these 3 files have been uploaded".

You would need to implement this logic yourself.

The uploads would trigger an AWS Lambda function. This function should check for the existence of the other two files. If they are present, it should continue processing. If they are not present, it should exit.

Upvotes: 2

Related Questions