CODEWITHSUNDEEP
python-3.xamazon-web-servicesamazon-ec2aws-lambda
akshay acharya
akshay acharya

Reputation: 163

How to trigger ec2 instance and a python script inside when a new object is added to s3 bucket?

I want to run a py script in my ec2 instance automatically whenever a new file is added to an s3 bucket. As of now, I have connected my bucket to a lambda function which is triggered with event notification whenever a file is added to the bucket. How do i further use this to trigger my ec2 instance to start executing the python file? I want to send that file from the bucket to the instance and the python program will use that file as input. I had read that i can trigger the lambda function when object is added. Now how do i use it or SQS to start my instance automatically and start running the python script inside it after the file has been sent from bucket to the instance?

Upvotes: 1

Views: 2234

Answers (3)

John Rotenstein
John Rotenstein

Reputation: 269101

There are several options:

  • Configure Amazon S3 to trigger an AWS Lambda function, and then do all of the processing inside the Lambda function, or
  • Have the EC2 instance running continuously. Configure S3 to push a message into an Amazon SQS queue. An app on the EC2 instance should continually long-poll the SQS queue to check if a file has been received. If so, the SQS message will contain the file details. or
  • Configure S3 to trigger an AWS Lambda function that then starts ("wakes up") an EC2 instance if it has been stopped. The Lambda function should push a message onto an SQS queue so that it is accessible to the instance (especially if multiple files have been created). Put a script on the EC2 instance in /var/lib/cloud/scripts/per-boot/, which will be run automatically when the instance starts. The script should trigger the app, the same as the 2nd option above, which reads from the SQS queue. Once it has emptied the queue, the script could stop the instance to minimize costs. For more details, see: Auto-Stop EC2 instances when they finish a task - DEV Community

Upvotes: 2

petrch
petrch

Reputation: 1968

S3 can generate an event, when a new object is created https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html

One type of event destination may be an SNS Topic https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html

One type of SNS subscription type is an HTTP/S POST https://docs.aws.amazon.com/sns/latest/api/API_Subscribe.html

So if you script can implement a simple HTTP listener/api, it will be triggered by SNS nearly instantly.

Upvotes: 0

lipek
lipek

Reputation: 106

uploading file to s3 bucket in an event which can trigger lambda function, as you wrote. The lambda may be python code itself - in this case there is no need to run it on ec2 instance. However you have time limit (15 mins) for execution. If this is not sufficient and the ec2 instance is normally stopped, you could start your ec2 instance using boto3 (you could use lambda code described here to do this and have the instance configured to run the python script on start (for example @reboot entry in /etc/crontab)

Upvotes: 0

Related Questions