CODEWITHSUNDEEP
pythonlinuxvisual-studio-codemultiprocessing
C. Daniel
C. Daniel

Reputation: 149

Why does my python multiprocessing script run on Windows but not on Linux?

I've successfully implemented a multiprocessed script on Windows, but the same script launches a "RuntimeError: already started" on linux and stops the execution. The script consists of the following "main.py" (omitted some part for readability):

from multiprocessing import freeze_support

if __name__ == '__main__':
    #MULTIPROCESSING STUFF
    freeze_support()

    #DO SOME STUFF

    #Call my multiprocessing-function in other module
    mod2.func(tileTS[0], label, areaconst)

And the "mod2.py" module:

import numpy as np
from multiprocessing import Pool
from functools import partial
import os, time

def func(ts, label, areaconst):
    #SETTING UP/LOADING SOME VARIABLES

    for idx in totImgs:            
        img_ = myList[idx]      

        p = Pool(2)
        result = p.map(  partial(_mp_avg, var1=var1_, img=img_), range(totObjs) ) 

        p.close()
        p.join()

        #MANAGE RESULTING VARIABLES

    return None


def _mp_avg(idx, img, var1):
    num = idx + 1
    arr = img[var1==num]
    if np.isnan(arr).any():
        return np.nan 
    else:
        return np.sum( arr )

This error is launched when the script executes the "Pool.map" function/class (dunno tbh). The same code works flawlessly on Windows.

I'm using Ubuntu 18.04 and launching the python 3.6.7 script from Visual Studio Code.

EDIT: added screenshot of runtime error(s) Terminal Error message

Upvotes: 6

Views: 3424

Answers (1)

C. Daniel
C. Daniel

Reputation: 149

As pointed out by @Darkonaut, Visual Studio Code uses ptvsd as debugger, which isn't fork-save (https://github.com/Microsoft/ptvsd/issues/1046#issuecomment-443339930). Since on linux the default process spawn method is "os.fork()", the script will generate a RuntimeError if executed from within VSCode. This will not happen on Windows. Solutions on Linux are:

  • Change start-method by inserting once the following line after the main function call: 

    multiprocessing.set_start_method("spawn")

  • Edit code with VSCode and launch from Terminal.

  • Change IDE.

  • Wait for fork-save debugger update, which is supposedly under work.

Check the following link for further information about the problem: https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

Upvotes: 8

Related Questions