CODEWITHSUNDEEP
pythonsubprocess
aeupinhere
aeupinhere

Reputation: 2983

Running a Python Subprocess

All,

I have read several threads on how to run subprocesses in python and none of them seem to help me. It's probably because I don't know how to use them properly. I have several methods that I would like to run at the same time rather than in sequence and I thought that the subprocess module would do this for me.

def services():
     services = [method1(),
            method2(),
            method3(),  
            mrthod4(),
            method5()]
     return services

def runAll():
    import subprocess
    for i in services():
        proc = subprocess.call(i,shell=True)

The problem with this approach is that method1() starts and method2() doesn't start until 1 finishes. I have tried several approaches including using subprocess.Popen[] in my services method with no luck. Can anyone lend me a hand on how to get methods 1-5 running at the same time?

Thanks, Adam

Upvotes: 1

Views: 2370

Answers (8)

j13r
j13r

Reputation: 2671

I had a similar problem recently, and solved it like this:

from multiprocessing import Pool
def parallelfuncs(funcs, args, results, bad_value = None):
    p = Pool()
    waiters = []
    for f in funcs:
        waiters.append(p.apply_async(f, args, callback = results.append))
    p.close()
    for w in waiters:
        if w.get()[0] == bad_value:
            p.terminate()
    return p

The nice thing is that the functions funcs are executed in parallel on args (kind of the reverse of map), and the result returned. The Pool of multiprocessing uses all processors and handles job execution.

w.get blocks, if that wasn't clear.

For your use case, you would call

results = []
parallelfuncs(services, args, results).join()
print results

Upvotes: 0

drone115b
drone115b

Reputation: 208

In python 3.2.x the concurrent futures module makes this sort of things very easy.

Upvotes: 1

Mikola
Mikola

Reputation: 9326

You need to use & to execute them asynchronously. Here is an example:

 subprocess.call("./foo1&", shell=True)
 subprocess.call("./foo2&", shell=True)

This is just like the ordinary unix shell.

EDIT: Though there are multiple, much better ways to do this. See the other answers for some examples.

Upvotes: 1

Corey Goldberg
Corey Goldberg

Reputation: 60644

subprocess.call() blocks until the process completes.

multiprocessing sounds more appropriate for what you are doing.

for example:

from multiprocessing import Process

def f1():
    while True:
        print 'foo'

def f2():
    while True:
        print 'bar'

def f3():
    while True:
        print 'baz'

if __name__ == '__main__':
    for func in (f1, f2, f3):
        Process(target=func).start()

Upvotes: 1

Fred Foo
Fred Foo

Reputation: 363807

By saying method1(), you're calling the function and waiting for it to return. (It's a function, not a method.)

If you just want to run a bunch of heavy-duty function in parallel and collect their result, you can use joblib:

from joblib import Parallel, delayed

functions = [fn1, fn2, fn3, fn4]

results = Parallel(n_jobs=4)(delayed(f)() for f in functions)

Upvotes: 1

Senthil Kumaran
Senthil Kumaran

Reputation: 56941

subprocess does not make the processes asynchronous. What you are trying to achieve can be done using multithreading or multiprocessing module.

Upvotes: 0

rafalotufo
rafalotufo

Reputation: 3942

Python threads are more appropriate to what you are looking for: http://docs.python.org/library/threading.html or even the multiprocessing module: http://docs.python.org/library/multiprocessing.html#module-multiprocessing.

Upvotes: 1

JMD
JMD

Reputation: 228

According to the Python documentation subprocess.call() waits for the command to complete. You should directly use the subprocess.Popen objects which will give you the flexibility you need.

Upvotes: 4

Related Questions