CODEWITHSUNDEEP
pythonmultithreadingmultiprocessingpython-multithreadingpython-multiprocessing
Ignasi
Ignasi

Reputation: 631

Paralelization python list comprehension not enhancing performance

Recently I have been trying to parallelize some list comprehensions to speed up my code but I found out that the parallelization lead to a worse time execution... can someone help me understand why?

my computer is an i7 4 cores 8 threads around 3GHz core speed and I am using python 2.7

Here you have an example of my code:

import numpy as np
import multiprocessing as mulpro
import itertools
d1 = 0.1;
d2 = 0.2;
data = range(100000) #Array of data
#example of list comprehension
data2 = [i + np.random.uniform(d1,d2) for i in data] #this is faster than the following
#example of multiprocessing
def parAddRandom(array):
    array = list(array)
    return (array[0] + np.random.uniform(array[1],array[2]))
pool = mulpro.Pool(processes=8)
data3 =  pool.map(parAddRandom, itertools.izip(data, itertools.repeat(d1), itertools.repeat(d2)))

I would expect the code to be faster by parallelization, as 8 cores are being used except from just 1, but it is not...

EDIT:

If I modify the code so the function parAddRandom only accepts one value then it is extremely faster...

import numpy as np
import multiprocessing as mulpro
import itertools
data = range(100000) #Array of data
#example of list comprehension
data2 = [i + np.random.uniform(d1,d2) for i in data] #Now this is not faster than the following
#example of multiprocessing
def parAddRandom(value):
    return (value + np.random.uniform(0.1,0.2))
pool = mulpro.Pool(processes=8)
data3 =  pool.map(parAddRandom, data)

But I still need to be able to modify the parameters "d1" and "d2" from the previous code...

Upvotes: 1

Views: 903

Answers (1)

Yoav Glazner
Yoav Glazner

Reputation: 8066

Because your function is small the overhead of the function call (and other multiprocessing machinery) is dominant )

import numpy as np
import timeit

d1 = 0.1;
d2 = 0.2;

def parAddRandom(array):

    return (array[0] + np.random.uniform(array[1],array[2]))


array = 45436, d1, d2
with_function_calling = timeit.timeit("parAddRandom(array)", globals=globals())

without_function_calling = timeit.timeit("array[0] + np.random.uniform(array[1],array[2])", globals=globals())

print ("function call adds {:0.2f}% overhead :(".format((100.0*with_function_calling/without_function_calling) - 100.0))

function alone call adds 18.59% overhead :(

My guess is that the other multiprocessing machinery add almost 100% in your example ...

If you want this to be effective you'll have to create a function that takes a bigger chunk each time.

Upvotes: 1

Related Questions