CODEWITHSUNDEEP
python-3.xpandasparallel-processing
PRO88
PRO88

Reputation: 437

Parallelize a function with data-frame as input

How I can parallelize the func1 in below code. It takes data-frame as input. func1 is the prototype of the function I am trying to parallelize. In real function have many data frames and series as input in the function. 

import pandas as pd

def func1(d,a):
    product = d.prod(axis=1)*a
    print(product)

#input 1
a1=2    
input1 = pd.DataFrame({'F':[2,3,4,5,1,2], 'E':[12,4,5,6,7,2], 'N':[2,7,6,5,4,3]}) 

#input 2
a2 = 0
input2 = pd.DataFrame({'F':[0,32,4,12,1,2], 'E':[1,4,5,0,7,2], 'N':[21,7,61,5,4,3]})

#input3
a3=100
input3 = pd.DataFrame({'F':[0,1,1,1,1,1], 'E':[1,12,5,110,7,2], 'N':[3,7,61,5,1,1]})

#call function
func1(input1,a1)
func1(input2,a2)
func1(input3,a3)

Upvotes: 2

Views: 185

Answers (1)

Renard Korzeniowski
Renard Korzeniowski

Reputation: 1105

Just use built-in library multiprocessing. I've edited your code to work on multiple cores simultaneously.

import pandas as pd
from multiprocessing import Pool, cpu_count

CORE_NUMBER = cpu_count()

def func1(d, a):
    product = d.prod(axis=1)*a
    print(product)

# input 1
a1=2    
input1 = pd.DataFrame({'F':[2,3,4,5,1,2], 'E':[12,4,5,6,7,2], 'N':[2,7,6,5,4,3]}) 

# input 2
a2 = 0
input2 = pd.DataFrame({'F':[0,32,4,12,1,2], 'E':[1,4,5,0,7,2], 'N':[21,7,61,5,4,3]})

# input 3
a3=100
input3 = pd.DataFrame({'F':[0,1,1,1,1,1], 'E':[1,12,5,110,7,2], 'N':[3,7,61,5,1,1]})

data = [(input1,a1),(input2,a2),(input3,a3)]
pool = Pool(CORE_NUMBER) 

# call function
pool.starmap(func1, data)
pool.close()
pool.join()

Upvotes: 2

Related Questions