CODEWITHSUNDEEP
pythonmultiprocessing
lisa1987
lisa1987

Reputation: 585

How do I run multiple subprocesses in parallel and wait for them to finish in Python

I am trying to migrate a bash script to Python.

The bash script runs multiple OS commands in parallel and then waits for them to finish before resuming, ie:

command1 &

command2 &

.

commandn &

wait

command

I want to achieve the same using the Python subprocess. Is this possible? How can I wait for a subprocess? call command to finish before resuming.

Upvotes: 28

Views: 53470

Answers (3)

Rainer Glüge
Rainer Glüge

Reputation: 1901

Noah Friedman's answer is nice but fails for some combinations of n and number of commands by skipping some final commands. I changed it to this, which i find much more readable:

import subprocess
commands = ['cmd1', 'cmd2', 'cmd3', 'cmd4', 'cmd5']
cpus = 2
while commands:
    batch = commands[:cpus]
    commands = commands[cpus:]
    procs = [subprocess.Popen(i, shell=True) for i in batch]
    for p in procs:
        p.wait()

Also note that this is not optimal, as a single hanging process holds back the next batch from being processed. The delay is actually noticeable. This is better:

import subprocess
from concurrent.futures import ProcessPoolExecutor

def my_parallel_command(command):
    subprocess.run(command, shell=True)

commands = ['cmd1', 'cmd2', 'cmd3', 'cmd4', 'cmd5']
cpus = 2
with ProcessPoolExecutor(max_workers = cpus) as executor:
    futures = executor.map(my_parallel_command, commands)

Upvotes: 2

Noah Friedman
Noah Friedman

Reputation: 151

Expanding on Aaron and Martin's answer, here is a solution that runs uses subprocess and Popen to run n processes in parallel:

import subprocess
commands = ['cmd1', 'cmd2', 'cmd3', 'cmd4', 'cmd5'] 
n = 2 #the number of parallel processes you want
for j in range(max(int(len(commands)/n), 1)):
    procs = [subprocess.Popen(i, shell=True) for i in commands[j*n: min((j+1)*n, len(commands))] ]
    for p in procs:
        p.wait()

I find this to be useful when using a tool like multiprocessing could cause undesired behavior.

Upvotes: 5

Martin Konecny
Martin Konecny

Reputation: 59611

You can still use Popen which takes the same input parameters as subprocess.call but is more flexible.

subprocess.call: The full function signature is the same as that of the Popen constructor - this functions passes all supplied arguments directly through to that interface.

One difference is that subprocess.call blocks and waits for the subprocess to complete (it is built on top of Popen), whereas Popen doesn't block and consequently allows you to launch other processes in parallel.

Try the following:

from subprocess import Popen
commands = ['command1', 'command2']
procs = [ Popen(i) for i in commands ]
for p in procs:
   p.wait()

Upvotes: 38

Related Questions