CODEWITHSUNDEEP
pythonamazon-s3amazon-ec2lambda
Joe
Joe

Reputation: 13091

Trigger Lambda function when a new object arrives in S3 bucket

I have S3 bucket named 'files'. Every day new file arrives there. Example:

/files/data-01-23-2017--11-33am.txt
/files/data-01-24-2017--10-28am.txt

How would I make a Lambda function and set a trigger to execute one shell script on EC2 when new file arrives? Example of new file is:

/files/data-01-25-2017--11-43am.txt

Command that I would want to execute on EC2 is (with parameter as new file name that just arrived):

python /home/ec2-user/jobs/run_job.py data-01-25-2017--11-43am.txt

Upvotes: 2

Views: 2756

Answers (1)

John Rotenstein
John Rotenstein

Reputation: 269101

Amazon S3 can be configured to trigger an AWS Lambda function when a new object is created. However, Lambda functions do not have access to your Amazon EC2 instances. It is not an appropriate architecture to use.

Some alternative options (these are separate options, not multiple steps):

  • Instead of running a command on an Amazon EC2 instance, put your code in the Lambda function (no EC2 instance required). (Best option!)
  • Configure Amazon S3 to push a message into an Amazon SQS queue. Have your code on the EC2 instance regularly poll the queue. When it receives a message, process the object in S3.
  • Configure Amazon S3 to send a message to an Amazon SNS topic. Subscribe an end-point of your application (effectively an API) to the SNS queue, so that it receives a message when a new object has been created.

Upvotes: 3

Related Questions