CODEWITHSUNDEEP
amazon-web-servicesamazon-ec2aws-lambda
DataNoob
DataNoob

Reputation: 205

Python pipeline on AWS Cloud

I have few python scripts which need to be executed in sequence on AWS Cloud so what are the best and simplest options? These script files are proof of concept so little bit dirty also but need to run overnight. Most of the script finishes within 10 mins but couple of them can take up to 1 hour running on a single core.

We do not have any servers like Jenkins, airflow etc...we are planning to use existing aws services.

Please let me know, Thanks.

Upvotes: 0

Views: 175

Answers (2)

Leonardo
Leonardo

Reputation: 9857

You didn't mention anything about which AWS resources your python scripts need to access or at least the purpose of the scripts, so it is difficult to provide a solution.

However a good option is to use AWS Batch.

Upvotes: 1

sfblackl
sfblackl

Reputation: 471

1) EC2 Instance (Manually controlled)

  • Upload your scripts to an S3 bucket Use default VPC
  • launch EC2 Instance
  • Use SSM Remote session to log in
  • Run AWS CLI (AWS S3 Sync to download from S3)
  • Run them Manually
  • stop instance when done.

To be clean, make a SH file (or master .py file) to do the work. If you want it to stop charging you money afterwards, add command to stop instance when complete.

Least amount of work

2) If you want to run scripts daily
- Script out the work above (include modifying the Autoscale group at end to go to one box) - Create an EC2 Auto Scale Group and launch it on a CRON job schedule.

It will start up, do the work, and then shut down and stop charging you.

3) Lambda
Pretty much like option 2, but AWS will do most of the work for you.
Either put all your scripts into one lambda..or put each script into its own lambda and have a master that does sync invoke of each script in the order you want.
You have a cloudwatch alarm trigger daily and does the work

I would say that if you are in POC mode, option 1 is best decision. It is likely closest to what you already do where you are currently executing. This is what @jarmod recommended already.

Upvotes: 1

Related Questions