Fargate, Lambda, another option ? what do I use for custom python job?

Question

I've just finished creating a python script that:

• imports some modules like requests, pandas and snowflake.connector
• reads data from an API
• manipulates the data using pandas
• saves the data as a CSV file locally
• Connects to Snowflake (database) and loads the data in to there
• when the script finishes, I don't need the CSV file any more (it will be recreated on the next run)

Now I need a good (serverless/cloud) service where I can run the script every hour.

I'm new to this sort of Python coding, let alone using serverless functionality.

I'm hoping there's something simple, like uploading my single .py file, setting the schedule (even a cron command) and sitting back and watching the data stream in to Snowflake.

Can someone recommend the best solution/configuration for this ?

Answer 1

There's no single answer to this.

You can use Amazon CloudWatch Events to trigger your Lambda functions using a cronjob expression: https://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-events.html

You can use the Serverless framework to make it easier: https://serverless.com/

There's a limit of 15 minutes of execution time on Lambda functions, if you need more than that then you could take a look at either https://aws.amazon.com/batch/ or https://aws.amazon.com/fargate/