CODEWITHSUNDEEP
pythonlinuxautomationcronscraper
user3482411
user3482411

Reputation: 1

How do I automate my scraper program that I wrote in python to run monthly?

I have written a Python program that scrapes information from a website using regex. My goal is to create a cron job to run this scraper each month.

I have gone into the Linux terminal, typed in crontab -e, and added to the bottom of the crontab file:

**

#!/usr/bin/python chmod +x
30 8 1 * * /home/pi/Nikita/The_Scraper/thescraper.py
PATH=/home/pi/Nikita/The_Scraper/thescraper.py
MAILTO="[email protected]"

**

I am wondering:

  1. If this is the correct text to include in the crontab file

  2. How to verify if my scraper program with run each month and the cron job I created is working

Upvotes: 0

Views: 482

Answers (1)

abraXxl
abraXxl

Reputation: 64

Where do I start:

1st: Cron starts the script via your shell. So /home/pi/Nikita/The_Scraper/thescraper.py have to have execute permissions.

2nd: PATH ist the name of the environment variable where the shell searches for your script if specified without PATH. It should contain only directories.

3rd: The crontab is read from top to bottom. It should be enough to use

MAILTO="[email protected]"
30 8 1 * * /home/pi/Nikita/The_Scraper/thescraper.py

This should run your script every 1st day of the month on 8:30.

The MAILTO setting specifies external mail address. You should have a properly configured MTA (program that delivers the mail) running. The content of that mail is STDOUT and STDERR of the mail.

To test you could specify a time some near time (5 minutes in the future) and see what happens. Also you can redirect the OUTPUT to a file then you see if the job has been run an what its OUTPUT was if sending a mail does not work.

30 8 1 * * /home/pi/Nikita/The_Scraper/thescraper.py > /some/test/file/were/your/user/can/write

If you have access to the system logs you are able to see if the cronjob has been executed.

Upvotes: 1

Related Questions