CODEWITHSUNDEEP
pythonmultiprocessingpython-3.9
Merlin
Merlin

Reputation: 25659

Multiprocessing changed in python 3.9 from 3.7 on MacOSx, how to fix?

Python 3.9 multiprocessing seems to run many extra loops over the whole file, while 3.7 didnt do this. The machine has 8 cores thus 8 loops. How do I fix this. This is not what I am expecting -- was the threading module also reworked, does this problem happen in that module also. This is the code:

 !/usr/bin/python
from   time            import strftime
import multiprocessing as mp

print (strftime("%H:%M:%S") + ' Run line5 ' )

def pullLast():  
    print(12345678998765432345678)
    return 1

def mprocULwork(uplist): # build a queue with tuple
    ULsyms = [e for e in uplist if e is not None] 
    p_size          = mp.cpu_count()
    pool            = mp.Pool(processes=p_size, maxtasksperchild=400,)
    pool_outputs    = pool.map(ULProcess, ULsyms)
    pool.close()    # no more tasks
    pool.join()     # wrap up current tasks
    del pool
    print (strftime("%H:%M:%S"), "Finished ULwork")

def ULProcess(symbol):
    pSet    = 0
    print(pSet,symbol)
    
if __name__ == '__main__': 
    pSet    = 1
    symlist = ["G","Y","S" ]
    ullist  = symlist

    global lastPriceDF
    lastPriceDF  = pullLast()

    mprocULwork(ullist)                ####### <<<<<<<<<<<<<<<<<<main entry
    print (strftime("%H:%M:%S"), "post")
   
print (strftime("%H:%M:%S"),  'Exiting....line last' )

This the output from python 3.7:

10:08:58 Run line5 
12345678998765432345678
0 G
0 Y
0 S
10:08:58 Finished ULwork
10:08:58 post
10:08:58 Exiting....line last

This is the output from 3.9:

10:20:44 Run line5 
12345678998765432345678
10:20:44 Run line5 
10:20:44 Exiting....line last
0 G
0 Y
0 S
10:20:44 Run line5 
10:20:44 Exiting....line last
10:20:44 Run line5 
10:20:44 Exiting....line last
10:20:44 Run line5 
10:20:44 Exiting....line last
10:20:44 Run line5 
10:20:44 Exiting....line last
10:20:44 Run line5 
10:20:44 Exiting....line last
10:20:44 Run line5 
10:20:44 Exiting....line last
10:20:44 Run line5 
10:20:44 Exiting....line last
10:20:44 Finished ULwork
10:20:44 post
10:20:44 Exiting....line last

Upvotes: 1

Views: 2132

Answers (1)

ShadowRanger
ShadowRanger

Reputation: 155506

Psychic debugging: You're on macOS. In 3.8, the multiprocessing module switched from using the 'fork' start method by default to the 'spawn' start method, because the high-level macOS system libraries are not fork-safe and can cause crashes. Part of how the 'spawn' method simulates forking is by reimporting the __main__ module (but with a different __name__ so a properly guarded module won't redo anything that's guarded).

If you want to roll the dice, go ahead and explicitly opt-in to the 'fork' start method with multiprocessing.set_start_method('fork'). More properly, fix your code to make it follow the multiprocessing guidelines for the 'spawn' (and 'forkserver') method.

Upvotes: 8

Related Questions