CODEWITHSUNDEEP
pythonwindowscmdparallel-processing
Dror Bogin
Dror Bogin

Reputation: 443

waiting for multiple Python processes to finish in Windows 7 CMD

I am trying to run multiple Python scripts in parallel in Windows 7 (and 10). I am running them all from another Python script, which performs more functions on the files the scripts are editing. I want the external script to wait until the other scripts are done running. I have tried start /w, but that made each script wait before closing the console window.

Essentially what I want to do is for Python to wait until the 3 processes are done. The last script is just a print("done"), and is meaningless for all I care. This is important for me to solve with 3 processes because I need to do the same thing with 30. (On a server, there are enough available threads.)

This is the CMD command I am trying to run.

os.system("start python node1.py & start python node2.py & start python node3.py && start /w printstatement.py")

Any suggestions?

Upvotes: 1

Views: 994

Answers (2)

Eryk Sun
Eryk Sun

Reputation: 34300

Use subprocess.Popen instead of os.system. You'll get 3 Popen instances that you can wait on. For example:

import subprocess

procs = [subprocess.Popen(['python', 'node{}.py'.format(n)]) 
            for n in range(1, 4)]

retcodes = [p.wait() for p in procs]

If you want separate console windows, like how CMD's start command works, then add the option creationflags=subprocess.CREATE_NEW_CONSOLE to the Popen call (Windows only). If you instead want separate consoles that don't create windows, use creationflags=CREATE_NO_WINDOW (0x08000000). In this case they still have console standard I/O; it's just not rendered to a window.

Upvotes: 6

vitaliy
vitaliy

Reputation: 464

Solution using asyncio:

import asyncio

commands = [
    'python node1.py',
    'python node2.py',
]

async def run_command(command):
    task = await asyncio.create_subprocess_exec(*command.split())
    await task.wait()

combined_task = asyncio.gather(*(run_command(command) for command in commands))
asyncio.get_event_loop().run_until_complete(combined_task)

Upvotes: 0

Related Questions