Run python web scraping script autonomously

Question

I have a web scraping script in python that goes to a particular website, logs in with my credentials and does some actions (like pressing some buttons etc). I want to run this script at specific intervals of time (say every 1 hour between 8am and 3pm).

I found that something like cron jobs would be able to take care of the time part. But I also would like it to run independent of my device. So I would like to know if it is possible to host the code on some server and make it run autonomously at the said times.

Answer 1

Sure it can run remotely, there are many options to do it. For example, you can use AWS lambda (the example uses a nodejs webserver but it can be done with a python script as well) and create a handler file that runs your python script, then use cloudwatch events to schedule it to run (it's edited in a cron-like manner).

The question is, what do you want to do with the results of the scraping. After all, you probably want to save it somewhere!

For that there are multiple options, a two I can think of right now are:

• save it into RDS (DB)
• save it as a file on S3