CODEWITHSUNDEEP
pythonlinuxbash
M1nt_zwy
M1nt_zwy

Reputation: 927

How to monitor a python process and restart it upon abnormal termination

Assume that there is a task.py,breaking due to memory overflow.How can i monitor this and restart it?

import time
while(1):
    print('.')
    # simulate breaks
    time.sleep(2)
    exit(0)

Thanks

Upvotes: 2

Views: 2544

Answers (3)

Jacques de Hooge
Jacques de Hooge

Reputation: 6990

You can use a watchdog. Make your worker process update a dummyfile say every 10 secs. Have another, completely independent, process check if the last access wasn't longer that say 20 secs ago. If it was, restart your worker process.

There are all kinds of nifty OS-dependent ways to do the same, but this low-tech one always works, even trivially over a network. Since you only measure time difference between two accesses, the clocks don't even have to be synchronized.

Upvotes: 4

Jahid
Jahid

Reputation: 22428

Something like this should work:

while ! /path/to/task.py; do
    echo 'restarting task...'
done

If task.py exits with non-zero exit status the loop will continue and run the script again. The loop will only break when task.py exits with 0.

If your program is errored and yield to non-zero exit at all time, this will end up being an infinite loop. So it's better to restrict the number of restart tries by a max_try value:

#!/bin/bash
max_try=100
count=1
while ! python /path/to/task.py; do
    ((count++)) # increment (Bashism)
    #count=$(expr $count + 1) # increment (portable)
    if [ $count -gt $max_try ]; then break; fi
    echo 'restarting task...'
done

Upvotes: 2

l0b0
l0b0

Reputation: 58828

If it actually runs out of memory it should get OOM killed. If you have another process which restarts it continuously (for example while true; do /path/to/my_script.py; done) it should get up and running again immediately.

Upvotes: 0

Related Questions