CODEWITHSUNDEEP
pythonparallel-processing
user991926
user991926

Reputation: 95

Parallel Python global name error

Hey im trying to run a simple code that adds a list of numbers together at the same time using Parallel Python

import sys, time
import pp
import numpy
x = numpy.arange(-20.0,20.0,0.5)
def function(raw_input):
    f = 0
    for i in numpy.arange(len(x)):
        f+=1
    a=raw_input[0]
    b=raw_input[1]
    c=raw_input[2]
    d=raw_input[3]
    print len(x)
    return (a+b+c+d)+f
# tuple of all parallel python servers to connect with
ppservers = ()
#ppservers = ("10.0.0.1",)

if len(sys.argv) > 1:
    ncpus = int(sys.argv[1])
    # Creates jobserver with ncpus workers
    job_server = pp.Server(ncpus, ppservers=ppservers)
else:
    # Creates jobserver with automatically detected number of workers
    job_server = pp.Server(ppservers=ppservers)
print "Starting pp with", job_server.get_ncpus(), "workers"

start_time = time.time()

# The following submits 4 jobs and then retrieves the results
puts = ([1,2,3,4], [3,2,3,4],[4,2,3,6],[2,3,4,5])

jobs = [(raw_input, job_server.submit(function,(raw_input,), (), ("numpy",))) for raw_input in puts]
for raw_input, job in jobs:
    print "Sum of numbers", raw_input, "is", job()

print "Time elapsed: ", time.time() - start_time, "s"
job_server.print_stats()

so basically i want it to add [1,2,3,4] together at the same time its adding [3,2,3,4],[4,2,3,6],[2,3,4,5]. And add "f" which is the length of x (which is 80) to all the answers. the out put should look like this:

Starting pp with 4 workers

Sum of numbers [1, 2, 3, 4] is 90

Sum of numbers [3, 2, 3, 4] is 92

Sum of numbers [4, 2, 3, 6] is 95

Sum of numbers [2, 3, 4, 5] is 94

Time elapsed: 0.394000053406 s

Job execution statistics:

job count | % of all jobs | job time sum | time per job | job server

     4 |        100.00 |       1.4380 |     0.359500 | local

Time elapsed since server creation 0.442999839783

the problem im having is that with x outside of "function" the shell comes back with global name 'x' not defined, but if you put x into the shell it returns the full array of x.

im confused why its clearly defined enough to give me back "x" when i put it in the shell but the job doesn't find x or anything else outside of the function definition.

Upvotes: 1

Views: 1511

Answers (1)

Peter de Rivaz
Peter de Rivaz

Reputation: 33509

I think the problem is that x is defined on the job server (which is the one running in the shell), but not in the job workers because the workers only have access to the variables in their inputs.

You should be able to fix it by passing in x as an additional argument when you submit the jobs.

As far as I can see, the calculation of f will always have the result of len(x) so you could simply pass in the value for f instead:

import sys, time
import pp
import numpy
x = numpy.arange(-20.0,20.0,0.5)
def function(raw_input,f):
    a=raw_input[0]
    b=raw_input[1]
    c=raw_input[2]
    d=raw_input[3]
    return (a+b+c+d)+f
# tuple of all parallel python servers to connect with
ppservers = ()
#ppservers = ("10.0.0.1",)

if len(sys.argv) > 1:
    ncpus = int(sys.argv[1])
    # Creates jobserver with ncpus workers
    job_server = pp.Server(ncpus, ppservers=ppservers)
else:
    # Creates jobserver with automatically detected number of workers
    job_server = pp.Server(ppservers=ppservers)
print "Starting pp with", job_server.get_ncpus(), "workers"

start_time = time.time()

# The following submits 4 jobs and then retrieves the results
puts = ([1,2,3,4], [3,2,3,4],[4,2,3,6],[2,3,4,5])

jobs = [(raw_input, job_server.submit(function,(raw_input,len(x)), (), ("numpy",))) for raw_input in puts]
for raw_input, job in jobs:
    print "Sum of numbers", raw_input, "is", job()

print "Time elapsed: ", time.time() - start_time, "s"
job_server.print_stats()

Upvotes: 1

Related Questions