CODEWITHSUNDEEP
pythonparallel-processingsubprocess
Hanfei Sun
Hanfei Sun

Reputation: 47071

Why this code doesn't work in parallel python

I tried to use pp(Parallel Python) like this:

import glob
import subprocess
import pp


def run(cmd): 
    print cmd
    subprocess.call(cmd, shell=True) 

job_server = pp.Server()
job_server.set_ncpus(8)
jobs = []
for a_file in glob.glob("./*"): 
    cmd = "ls" 
    jobs.append(job_server.submit(run, (cmd,))) 
for j in jobs: 
    j()

But encountered such an error that subprocess.call is not a global name.

An error has occured during the function execution
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pp-1.6.1-py2.7.egg/ppworker.py", line 90, in run
    __result = __f(*__args)
  File "<string>", line 3, in run
NameError: global name 'subprocess' is not defined

I've imported subprocess, why can't it be used here?

According to abarnert's suggestion, I changed my code to this:

import glob
import pp
def run(cmd): 
    print cmd
    subprocess.call(cmd, shell=True) 
job_server = pp.Server()
job_server.set_ncpus(8)
jobs = []
for a_file in glob.glob("./*"): 
    cmd = "ls" 
    jobs.append(job_server.submit(run, (cmd,),modules=("subprocess",))) 
for j in jobs: 
    j()

But it still doesn't work, it complains like this:

Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
   self.run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.6/dist-packages/pp-1.6.1-py2.6.egg/pp.py", line 721, in _run_local
    job.finalize(sresult)
UnboundLocalError: local variable 'sresult' referenced before assignment

Upvotes: 1

Views: 4291

Answers (2)

abarnert
abarnert

Reputation: 365945

The documentation explains this pretty well, and each example shows you how to deal with it.

Among the params of the submit method is "modules - tuple with module names to import". Any modules you want to be available in the submitted job has to be listed here.

So, you can do this:

jobs.append(job_server.submit(run, (cmd,), (), ('subprocess',)))

Or this:

jobs.append(job_server.submit(run, (cmd,), modules=('subprocess',)))

Upvotes: 2

mikebabcock
mikebabcock

Reputation: 791

Sorry, untested, but did you try:

from subprocess import call

Inside the 'run' function?

And then use "call" instead of "subprocess.call" ? That would make 'call' local to the function but accessible.

Upvotes: 1

Related Questions