CODEWITHSUNDEEP
pythonpython-multiprocessingentry-point
Shivkumar kondi
Shivkumar kondi

Reputation: 6762

Python multiprocessing with starmap and issue with __main__

I was trying to make small multiprocessing with multiple parameters

TaskType-1:

import multiprocessing  as mp
import pandas as pd
import os,sys
print("Libs Loaded..!!! ")
listoflists = [[1,2,3,4],[4,5,6,7],[7,8,9,10],[10,11,13,14]]
listoftuples = [tuple(i) for i in listoflists]
print("Length of Tuples : ",len(listoftuples))

def map_function(combo):
    a = combo[0]
    b = combo[1]
    c = combo[2]
    print((a + b + c))
    return (a + b + c)

def doit():
    try:
        print("IN main")
        p = mp.Pool(processes=2)
        # results= p.map(map_function, listoftuples)
        results = p.starmap(map_function,listoftuples)
        print(results)
        print("Done!!")
    except Exception as e:
        exc_type, exc_obj, exc_tb = sys.exc_info()
        fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
        print(exc_type, fname, exc_tb.tb_lineno)

# if '__name__' == '__main__': 
doit()     # running without entry point

Error for TaskType-1:

Libs Loaded..!!!
Length of Tuples :  4
IN main
Libs Loaded..!!!
Length of Tuples :  4
IN main
<class 'RuntimeError'> testformp.py 20
<class 'TypeError'> testformp.py 22
Libs Loaded..!!!
Length of Tuples :  4
IN main
<class 'RuntimeError'> testformp.py 20

Not sure this runtime Error and why it goes in into doit() function multiple times. Multipeprocessing is defined within this function but here it is calling the parent function again and again..Not sure what I am missing here to understand?

TaskType-2:

if '__name__' == '__main__':
    doit()  # running from entry point

output for TaskType-2:

Libs Loaded..!!!
Length of Tuples :  4

It shows no errors nor it performs any task inside. Why this is so?

Upvotes: 0

Views: 665

Answers (1)

quamrana
quamrana

Reputation: 39354

Making the two changes I suggested I end up with this:

import multiprocessing  as mp
import os,sys
#print("Libs Loaded..!!! ")
listoflists = [[1,2,3,4],[4,5,6,7],[7,8,9,10],[10,11,13,14]]
listoftuples = [tuple(i) for i in listoflists]
#print("Length of Tuples : ",len(listoftuples))

def map_function(*combo):
    a = combo[0]
    b = combo[1]
    c = combo[2]
    #print((a + b + c))
    return (a + b + c)

def doit():
    try:
        print("IN main")
        p = mp.Pool(processes=2)
        #results= p.map(map_function, listoftuples)
        results = p.starmap(map_function,listoftuples)
        print(results)
        print("Done!!")
    except Exception as e:
        exc_type, exc_obj, exc_tb = sys.exc_info()
        fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
        print(exc_type, fname, exc_tb.tb_lineno)

if __name__ == '__main__':
    doit()     # running without entry point

And the output is this:

IN main
[6, 15, 24, 34]
Done!!

Upvotes: 1

Related Questions