How to get notified when a script didn't run

Question

I have a bash script that runs on a machine (called lazyguy) that is usually turned off. Every night, lazyguy wakes up at a known time programmed in the BIOS, does a bunch of processing and inserts data to its Postgres database. Then, it goes to sleep (database and all).

How can get notified when the script did NOT run? Ideally, I'd like to get an email notification if nothing was inserted to the database or if the script exited with a non-zero return value. I definitely need to check for both conditions in case lazyguy ignored his alarm and slept in..again.

I have another machine (taskmaster) that is always running but has limited resources. Some thoughts:

• Have lazyguy send some sort of message to taskmaster to indicate that he finished his chores (maybe with a count of rows inserted). Then, taskmaster will check for his report everyday at a certain time. If it's not there, email me and I'll send the goons to beat up lazyguy.

• Have taskmaster wakeup lazyguy directly with a WakeOnLan packet, wait two minutes, run the script via ssh, and check the number of rows in the database for the day. If anything goes bad, email me.

Which approach is better? Is there an easier way to do this? Which technologies would you use? I'm thinking maybe some mix of crontab, mailx, logger, and maybe rsyslog.

Answer 1

I'm surprised no one has mentioned this yet: When a script run from cron exits abnormally, it sends an email to the mail address specified in /etc/crontab. It includes all output of the script.

Answer 2

You can send your self an email through a command line with the mail command. Use this link http://www.amirwatad.com/blog/archives/2009/03/21/send-email-from-the-command-line-using-gmail-account/ to set it up. you can set a special account for this.

Answer 3

i would prefer your first, more decentralized method (e.g. lazyguy would do some request on a simple HTTP server running on taskmaster after successful job, the server would update the timestamp stored on the disk, which would be checked by cron script), but the the second one has an advantage - when you decide to change the lazyguy scheduling (and so a check scheduling), you would do it from a single place - taskmaster.

Answer 4

if lazyguy can print out something to a file after all of his chores are done, then it would be great.

• • • • • rm -rf /tmp/lazyguy.log; /lazyguy.sh > /tmp/lazyguy.log

then have another script run at another time to check if lazyguy did finished, still running or did not run at all.

to check your if lazyguy you can do something like

checkamdump(){
while ps ax | grep -v grep | grep "lazyguy.sh" > /dev/null
do
sleep $SLEEPTIMEOUT
done
}


Also, you can check lazyguy.log by adding this to your script

FILESIZE=stat -c %s /tmp/lazyguy.log
if [ $FILESIZE -gt 0 ]
then
mailx -s "Lazyguy is really lazy" alerts@main.com < /tmp/lazyguy.log
fi
Hope this helps, add your comment if you have any other questions