CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-data-pipeline
ScottieB
ScottieB

Reputation: 4052

What's the best way to run a python script daily?

I have a python script that connects to Redshift, executes a series of SQL commands, and generates a new derived table.

But for the life of me, I can't figure out a way to have it automatically run every day.

I've tried AWS Data Pipeline but my shell script won't run the first copy statement. I can't get Lambda or Glue to work because my company's IAM policies are restrictive. Airflow seems like overkill to just run a single python script daily.

Any suggestions for services to look into?

Upvotes: 4

Views: 9787

Answers (4)

Moe
Moe

Reputation: 2842

use a cron job on an ec2 instance or set up a scheduled event to invoke your aws python lambda function http://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-events.html

Upvotes: 1

Joshua Guttman
Joshua Guttman

Reputation: 74

If you are using AWS Glue or have some other reason to install a development endpoint, you can use Apache Zeppelin to run any code from any language (if you have the jar files) on a schedule based on a cron command. Here's the notebook I use to run Redshift nightly maintenance:

Redshift Maintenance in a Zeppelin notebook

Upvotes: 1

Mike Robins
Mike Robins

Reputation: 1773

I use a scheduled task on Windows. Either enter it using the GUI or the at command.

Upvotes: 1

Jack Ryan
Jack Ryan

Reputation: 1318

Cron job?

00 12 * * * /home/scottie/bin/my_python_script.py

Run my_python_script.py at the top of the hour (0th minute), at noon, every day.

Upvotes: 7

Related Questions