CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-ec2aws-lambdajob-scheduling
sk3145
sk3145

Reputation: 174

Run python code on AWS service periodically

I need to run some python code on aws platform periodically(probably once a day). Program job is to connect to S3, download some files from bucket, do some calculations, upload results back to S3. This program runs for about 1 hour so I cannot make use of Lambda function as it has a maximum execution time of 900s(15mins).

I am considering to use EC2 for this task. I am planning to setup python code into a startup and execute it as soon as the EC2 instance is powered on. It also shuts down the instance once the task is complete. The periodic restart of this EC2 will be handled by lambda function.

Though this a not a best approach, I want to know any alternatives within aws platform(services other than EC2) that can be best of this job.

Since

Upvotes: 2

Views: 1318

Answers (2)

tatlar
tatlar

Reputation: 3176

Using EC2 or Fargate may be significant overkill. Creating a simple AWS Glue job triggered by a Lambda function (running once per day) to do this (pull from S3, open selected files (if required), do some calculations on the files contents, then push results back to S3) using Python and the AWS boto3 library (and other standard Python file-reading libs if necessary) is most likely your easiest route.

See this SO question for an example and solution.

Good luck!

Upvotes: -1

Lucas Barbosa
Lucas Barbosa

Reputation: 138

If you are looking for other solutions other than lambda and EC2 (which depending on the scenario it fits) you could use ECS (Fargate).

It's a great choice for microservices or small tasks. You build a Docker image with your code (Python, node, etc...), tag it and then you push the image to AWS ECR. Then you build a cluster for that and use the cloudwatch to schedule the task with Cloudwatch or you can call a task directly either using the CLI or another AWS resource.

  • You don't have time limitations like lambda
  • You don’t also have to setup the instance, because your dependencies are managed by Dockerfile
  • And, if needed, you can take advantage of the EBS volume attached to ECS (20-30GB root) and increase from that, with the possibility of working with EFS for tasks as well.

I could point to other solutions, but they are way too complex for the task that you are planning and the goal is always to use the right service for the job

Hopefully this could help!

Upvotes: 2

Related Questions