CODEWITHSUNDEEP
pythonwindowsasynchronous
Rafał Dowgird
Rafał Dowgird

Reputation: 45171

Python on Windows - how to wait for multiple child processes?

How to wait for multiple child processes in Python on Windows, without active wait (polling)? Something like this almost works for me:

proc1 = subprocess.Popen(['python','mytest.py'])
proc2 = subprocess.Popen(['python','mytest.py'])    
proc1.wait()
print "1 finished"
proc2.wait()
print "2 finished"

The problem is that when proc2 finishes before proc1, the parent process will still wait for proc1. On Unix one would use waitpid(0) in a loop to get the child processes' return codes as they finish - how to achieve something like this in Python on Windows?

Upvotes: 25

Views: 23199

Answers (6)

Thomas Grainger
Thomas Grainger

Reputation: 2429

you can use psutil

import psutil

with psutil.Popen(["python", "mytest.py"]) as proc1, psutil.Popen(
    ["python", "mytest.py"]
) as proc2:
    gone, alive = psutil.wait_procs([proc1, proc2], timeout=3)

'gone' and 'alive' are lists indicating which processes are gone and which ones are still alive.

Optionally you can specify a callback which gets invoked every time one of the watched processes terminates:

def on_terminate(proc):
    print "%s terminated" % proc

gone, alive = psutil.wait_procs(ls, timeout=3, callback=on_terminate)

Upvotes: 1

Giampaolo Rodolà
Giampaolo Rodolà

Reputation: 13076

You can use psutil:

>>> import subprocess
>>> import psutil
>>> 
>>> proc1 = subprocess.Popen(['python','mytest.py'])
>>> proc2 = subprocess.Popen(['python','mytest.py'])    
>>> ls = [psutil.Process(proc1.pid), psutil.Process(proc2.pid)]
>>>
>>> gone, alive = psutil.wait_procs(ls, timeout=3)

'gone' and 'alive' are lists indicating which processes are gone and which ones are still alive.

Optionally you can specify a callback which gets invoked every time one of the watched processes terminates:

>>> def on_terminate(proc):
...     print "%s terminated" % proc
...
>>> gone, alive = psutil.wait_procs(ls, timeout=3, callback=on_terminate)

Upvotes: 2

Ted Mielczarek
Ted Mielczarek

Reputation: 3967

Building on zseil's answer, you can do this with a mix of subprocess and win32 API calls. I used straight ctypes, because my Python doesn't happen to have win32api installed. I'm just spawning sleep.exe from MSYS here as an example, but clearly you could spawn any process you like. I use OpenProcess() to get a HANDLE from the process' PID, and then WaitForMultipleObjects to wait for any process to finish.

import ctypes, subprocess
from random import randint
SYNCHRONIZE=0x00100000
INFINITE = -1
numprocs = 5
handles = {}

for i in xrange(numprocs):
    sleeptime = randint(5,10)
    p = subprocess.Popen([r"c:\msys\1.0\bin\sleep.exe", str(sleeptime)], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False)
    h = ctypes.windll.kernel32.OpenProcess(SYNCHRONIZE, False, p.pid)
    handles[h] = p.pid
    print "Spawned Process %d" % p.pid

while len(handles) > 0:
    print "Waiting for %d children..." % len(handles)
    arrtype = ctypes.c_long * len(handles)
    handle_array = arrtype(*handles.keys())
    ret = ctypes.windll.kernel32.WaitForMultipleObjects(len(handle_array), handle_array, False, INFINITE)
    h = handle_array[ret]
    ctypes.windll.kernel32.CloseHandle(h)
    print "Process %d done" % handles[h]
    del handles[h]
print "All done!"

Upvotes: 7

user23475
user23475

Reputation: 41

Twisted on Windows will perform an active wait under the covers. If you don't want to use threads, you will have to use the win32 API to avoid polling. Something like this:

import win32process
import win32event

# Note: CreateProcess() args are somewhat cryptic, look them up on MSDN
proc1, thread1, pid1, tid1 = win32process.CreateProcess(...)
proc2, thread2, pid2, tid2 = win32process.CreateProcess(...)
thread1.close()
thread2.close()

processes = {proc1: "proc1", proc2: "proc2"}

while processes:
    handles = processes.keys()
    # Note: WaitForMultipleObjects() supports at most 64 processes at a time
    index = win32event.WaitForMultipleObjects(handles, False, win32event.INFINITE)
    finished = handles[index]
    exitcode = win32process.GetExitCodeProcess(finished)
    procname = processes.pop(finished)
    finished.close()
    print "Subprocess %s finished with exit code %d" % (procname, exitcode)

Upvotes: 4

tzot
tzot

Reputation: 96071

It might seem overkill, but, here it goes:

import Queue, thread, subprocess

results= Queue.Queue()
def process_waiter(popen, description, que):
    try: popen.wait()
    finally: que.put( (description, popen.returncode) )
process_count= 0

proc1= subprocess.Popen( ['python', 'mytest.py'] )
thread.start_new_thread(process_waiter,
    (proc1, "1 finished", results))
process_count+= 1

proc2= subprocess.Popen( ['python', 'mytest.py'] )
thread.start_new_thread(process_waiter,
    (proc2, "2 finished", results))
process_count+= 1

# etc

while process_count > 0:
    description, rc= results.get()
    print "job", description, "ended with rc =", rc
    process_count-= 1

Upvotes: 19

Glyph
Glyph

Reputation: 31910

Twisted has an asynchronous process-spawning API which works on Windows. There are actually several different implementations, many of which are not so great, but you can switch between them without changing your code.

Upvotes: 5

Related Questions