CODEWITHSUNDEEP
pythonparallel-processing
Shoeb Ahmed
Shoeb Ahmed

Reputation: 25

How do I run a single function over a loop parallely? Python 2.7.12

I'm trying to parallelize a program which uses a single function in a for loop which updates a global list/variable parallely. How do I go about this and how can I pass values to the function?

The sample code looks like this,

#python 2.7.12
import random
sum=0

#sample function to take a random integer between 0 to i and add it to global variable sum
def hello(i):
    print "Thread Id :",i   #instead of i it should be the threadID
    sum=sum+random.randrange(0,i)

#main function  
i=1
for i in range(10): 
    hello(i)        #how to parallelize this function over a loop?

Edit 1 : Tried using the Process from multiprocessing but don't know how to pass values to the functions and how to run it parallely over a loop.

from multiprocessing import Process
sum=0 

def Hello(): 
   print 'hello: starting' 
   sum=sum+i    #Don't know how to pass values here 
   print 'hello: finishing' 

if name == 'main': 
   p1 = Process(target=func1) 
   p1.start() 
   p1.join()
   print sum

Upvotes: 1

Views: 788

Answers (2)

pscuderi
pscuderi

Reputation: 1554

You can use multiprocessing.dummy.Pool, which takes the function followed by your parameters (see below).

You'll also need to worry about synchronization on your global variable (see below for an example of how to use Lock).

Also, unless you use "global sum" the sum inside your function is referring to a local sum variable (see below for global sum example).

threading.current_thread() gives you the thread id.

#python 2.7.12
from multiprocessing.dummy import Pool as ThreadPool
import threading
import random

lock = threading.Lock()

sum = 0

#sample function to take a random integer between 0 to i and add it to global variable sum
def hello(i):
    if (i == 0):
        return
    global sum
    print threading.current_thread()   #instead of i it should be the threadID
    r = random.randrange(0,i)
    lock.acquire()
    try:
        sum = sum + r
    finally:
        lock.release()

ThreadPool().map(hello, list(range(1, 11)))

print sum

Upvotes: 2

Michael Robellard
Michael Robellard

Reputation: 2358

Pass the parameters as a tuple to the args kwarg of the Process function: Here is the example from the docs:

if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()

https://docs.python.org/2/library/multiprocessing.html

Also what John Gordon said in his comment is true too. You will need to have the master process (which will be running the loop) handle the summing. What you are basically trying to do is a simple map/reduce job.

The map part is your hello function and the reduce is the sum. There are lots of examples of doing map/reduce jobs in python.

Look at using a Pipe or Shared Memory for handling the sum, both are detailed in the multiprocessing docs mentioned above.

Upvotes: 0

Related Questions