CODEWITHSUNDEEP
pythonjobssubprocessparallel-processing
nabizan
nabizan

Reputation: 3365

python parallel programming

hi i need some guidelines how to write programm that executes other python programm but in maximum for example 6 times at once and always try to be 6 process be running even if one ends up

also i would like to know what is happening to those processes right know but i dont want to wait to any process to finish

what is the pid of just created process? and its still running? or there has been an error? or it finish sucessfully?

some job manager ...

import subprocess

def start():
    proc = {}
    for i in range (0,6):
        proc[i] = subprocess.Popen(
            ['python', 'someprogramm.py', '--env', 'DEVELOPMENT', '-l'],
            shell = True, 
            stdout = subprocess.PIPE, 
            stderr = subprocess.STDOUT
        )


if __name__ == '__main__':
    start()

Upvotes: 2

Views: 2055

Answers (4)

S.Lott
S.Lott

Reputation: 391846

Use celery.

Upvotes: 4

Cédric Julien
Cédric Julien

Reputation: 80761

Maybe you can try with the multiprocessing module, it can handles pool of worker processes that seems similar to what you try to achieve.

Upvotes: 1

Ikke
Ikke

Reputation: 101231

You can use the poll() method to check if a process is still running. You'd have to loop through each process and check if it runs, and otherwise, run a new process.

Upvotes: 0

MattH
MattH

Reputation: 38247

Have a look at supervisord. It sounds like it will do what you want.

Upvotes: 1

Related Questions