CODEWITHSUNDEEP
pythonlinuxsubprocessmultiprocessing
Vor
Vor

Reputation: 35149

How to stop all child processes spawnedby a single script using subprocess

I need to run a unittest on part of my package, and the best thing for me to do this is to control launching it and then killing all processes spawned by multiprocessing module.

Here is what I'm talking about:

test.py

import logging
import multiprocessing
import time
import random

log = logging.getLogger(__name__)

def start_consumers(conf, worker_count=5):
    manager = WorkerManager(conf, worker_count)
    manager.start()


class WorkerManager():
    def __init__(self, conf, worker_count=5):
        self.workers = []
        for num in range(worker_count):
            self.workers.append(WorkHandler(conf))

    def start(self):
        for worker in self.workers:
            worker.daemon = True
            worker.start()

        print 'started'
        for worker in self.workers:
            worker.join()

class WorkHandler(multiprocessing.Process):

    def __init__(self, conf, *args, **kwargs):
        super(WorkHandler, self).__init__(*args, **kwargs)
        self.conf = conf
        self.name = str(random.randint(0,100))

    def run(self):
        while True:
            print self.conf['foo'], self.name
            time.sleep(3)

if __name__ == "__main__":
    conf = {'foo': 'bar'}
    start_consumers(conf)

Now if I run this test from linux terminal, I can see print statements, also if i do: ps aux | grep python I see all child processes where spawned:

sergey    4081  0.0  0.1  71684 13164 pts/3    S+   08:58   0:00 python
sergey    4108  0.3  0.0  39092  6472 pts/3    S+   08:59   0:00 python test.py
sergey    4109  0.0  0.0  39092  4576 pts/3    S+   08:59   0:00 python test.py
sergey    4110  0.0  0.0  39092  4568 pts/3    S+   08:59   0:00 python test.py
sergey    4111  0.0  0.0  39092  4576 pts/3    S+   08:59   0:00 python test.py
sergey    4112  0.0  0.0  39092  4584 pts/3    S+   08:59   0:00 python test.py
sergey    4113  0.0  0.0  39092  4580 pts/3    S+   08:59   0:00 python test.py
sergey    4115  0.0  0.0  13588   944 pts/7    S+   08:59   0:00 grep --color=auto python

Now if I'm trying to run the same test.py with subprocess everything works fine up to the point until I need to kill them all:

>>> import subprocess as s
>>> p = s.Popen(['python', 'test.py'], stdout=s.PIPE)

The Question:

What is the most elegant way to do something with this variable p in order to achieve similar behaviour when I press Ctrl+C in terminal?
In other words I want to achive a result similar to pkill -f "test.py"

UPDATE:

p.kill() or p.terminate() doesn't do what I need:

Here is the result after I do eather of them:

sergey    4438  0.0  0.0      0     0 pts/3    Z+   09:16   0:00 [python] <defunct>
sergey    4439  0.0  0.0  39092  4572 pts/3    S+   09:16   0:00 python test.py
sergey    4440  0.0  0.0  39092  4568 pts/3    S+   09:16   0:00 python test.py
sergey    4441  0.0  0.0  39092  4576 pts/3    S+   09:16   0:00 python test.py
sergey    4442  0.0  0.0  39092  4580 pts/3    S+   09:16   0:00 python test.py
sergey    4443  0.0  0.0  39092  4580 pts/3    S+   09:16   0:00 python test.py

Upvotes: 2

Views: 3014

Answers (2)

jfs
jfs

Reputation: 414745

You could try to create a new session and kill all descendant processes using os.killpg():

import os
import signal
from subprocess import Popen

p = Popen('python test.py', shell=True, preexec_fn=os.setsid)
# ... later
os.killpg(p.pid, signal.SIGTERM) # send signal to the process group

See How to terminate a python subprocess launched with shell=True.

Upvotes: 1

aruisdante
aruisdante

Reputation: 9105

I believe what you're looking for is the subprocess.terminate() and subprocess.kill() methods of subprocess objects.

You will likely have to combine that with a method registered with atexit that iterates across your list of subprocesses and terminates them. EX:

def terminate_children(children):
    for process in children:
        process.terminate()

...
# Somewhere else in your code
children = [s.Popen(['python', 'test.py'], stdout=s.PIPE) for i in range(number_processes)] # Spools up number_processes child processes
atexit.register(terminate_children, children) # where children is a list of subprocesses

This will terminate all child processes gracefully when you terminate the parent process gracefully.

If you're trying to kill the processes from a completely separate script with no direct reference to the child code, see here. You'll basically want to look at the python os process management methods

Upvotes: 2

Related Questions