os.system can start but cannot kill a Python process

Question

I am unable to stop a python script from running using os.system() but I am able to make it run with os.system(). When I run the same command on the terminal to kill the process, it works.

I have this code snippet from my web_plants.py script to deploy it in html using Flask:

@app.route("/auto/water/<toggle>")
def auto_water(toggle):
    running = False
    if toggle == "ON":
        templateData = template(text = "Auto Watering On")
        for process in psutil.process_iter():
            try:
                if process.cmdline()[1] == 'auto_water.py':
                    templateData = template(text = "Already running")
                    running = True
            except:
                pass
        if not running:
            os.system("python auto_water.py")
    else:
        templateData = template(text = "Auto Watering Off")
        os.system("pkill -f water.py")

    return render_template('main.html', **templateData)

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=80, debug=True)

I have a water.py script that contains the auto_water function for the button on my html page. But the function works fine, so I suppose that's not an issue.

def auto_water(pump_pin = 38):
    init_output(pump_pin)
    print("Reading Soil Humidity values from ADS1115")

    try:
        while True:
            m = adc.read_adc(0, gain=GAIN)
            print('Moisture Level:{0:>6}'.format(m))
            time.sleep(1)
            if m>23000: 
                #GPIO.output(pump_pin, GPIO.LOW)
                print "relay on"
            else:
                #GPIO.output(pump_pin, GPIO.HIGH)
                print "relay off"
    except KeyboardInterrupt:
        GPIO.cleanup()
    except Exception as e:
        print (e)

And I have this auto_water.py file that imports water and calls the auto_water() function to run it. The purpose of using this file is to be able to kill the file while it is running when I click on the "Stop" button.

import water

#From the water.py script, call the auto_water function and run it as a process

if __name__ == "__main__":
    water.auto_water()

The script keeps running and I have to kill it using sudo pkill -f water.py from the terminal in order to make it stop. But the same command doesn't work on the web_plants.py file, which I have used to stop it when I press the "Stop" button. However when I kill it, the webpage does register it and displays the text "Auto Watering OFF" as in the following lines from the web_plants.py file:

templateData = template(text="Auto Watering Off")
os.system("pkill -f water.py") 

Answer 1

Can you use os.kill() instead?

import os
import signal

os.kill(pid, signal.SIGTERM)

If you don't have the pid readily available, use subprocess.check_output to acquire it. (Use subprocess.run instead if you're using Python 3.5+.)

import os
import signal
import subprocess

def get_water_pid():
    return int(subprocess.check_output(['pgrep', '-f', 'water.py']).strip())

def kill_water_process():
    pid = get_water_pid()
    os.kill(pid, signal.SIGTERM)

Answer 2

The system call blocks the execution. The function wont return anything back to the browser until the execution is complete.

Try this:

os.system("python auto_water.py&") 

& spawns the process and return immediately.

Answer 3

Assuming the script is only ever started by your server, you can store the script's PID when you start it, and then use that to kill it.

It looks like your existing code is not spinning up the script in the background; it's leaving a Flask thread tied up forever waiting on the os.system to return. That's a bit weird, but I'll do the same thing for now.

Also, I'll just write bare functions, which you can Flask up as appropriate, and store the PID in a global, which you can replace with whatever persistent storage you're using.

autowater_pid = None

def start():
    global pid
    p = subprocess.Popen(["python", "auto_water.py"])
    pid = p.pid
    p.wait()

def stop():
    global pid
    if pid is not None:
        os.kill(pid)
        pid = None

---

You can expand on this in many ways. For example, if we already have a pid, we can skip starting a new copy. Or we can even check that it's still running, and start a new one if it's died somehow:

def start():
    global pid
    if pid is not None:
        try:
            os.kill(pid, 0) # check that it's still running
            return
        except ProcessLookupError:
            pass
    p = subprocess.Popen(["python", "auto_water.py"])
    pid = p.pid
    p.wait()

---

But back to that problem with waiting forever for the process to finish. Not only are you permanently tying up a thread, you're also making it so that that if the Flask process goes down for any reason (upgrade, load balancer, unhandled exception, whatever), the script will be killed by the OS. You should consider daemonizing the script in some way. That's way too big a topic to include in a side note to an answer, but you should be able to find enough information from searching to get started (or, at least, to come up with a good separate question to ask on SO).

---

Another alternative—and the one more commonly used by daemons—is to have the script itself (or a daemonizing wrapper) write its own pid to a file in a known location at startup. Like this:

with open('/var/run/auto_water.pid', 'w') as f:
    f.write(f'{os.getpid()}\n')

And you can os.remove('/var/run/auto_water.pid') on exit.

And then your server can read that number out of the file to know the PID to kill. And again, you can put in code to handle the case where the process died without erasing its pidfile, etc.

However, to do it properly, you're going to want to read up on locking pidfiles, the right behavior for handling unexpected existing ones, etc.

Or, better, just get a pidfile-handling library off PyPI.

Or, maybe even better, get a daemon-handling library that does the daemonizing and the pidfile and the error handling all in one.