CODEWITHSUNDEEP
pythonnumpypytorchdata-science
Артем Дачевский
Артем Дачевский

Reputation: 403

How to fix multiprocessing problems in python in windows10

I try to use this tutorial to train my own car model recognition model: https://github.com/Helias/Car-Model-Recognition. And i want to use coda and my gpu perfomance to enhance training speed (preprocesssing step was completed without any errors).But when I try to train my model, I've got the following errors:

######### ERROR #######

        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.


######### batch #######
Traceback (most recent call last):
  File "D:\Car-Model-Recognition\main.py", line 78, in train_model


######### ERROR #######
[Errno 32] Broken pipe
    for i, batch in enumerate(loaders[mode]):


######### batch #######  File "C:\Program Files\Python37\lib\site-packages\torch\utils\data\dataloader.py", line 279, in __iter__

    return _MultiProcessingDataLoaderIter(self)
Traceback (most recent call last):
  File "C:\Program Files\Python37\lib\site-packages\torch\utils\data\dataloader.py", line 719, in __init__
  File "main.py", line 78, in train_model
    w.start()
  File "C:\Program Files\Python37\lib\multiprocessing\process.py", line 112, in start
    for i, batch in enumerate(loaders[mode]):
  File "C:\Program Files\Python37\lib\site-packages\torch\utils\data\dataloader.py", line 279, in __iter__
    self._popen = self._Popen(self)
  File "C:\Program Files\Python37\lib\multiprocessing\context.py", line 223, in _Popen
    return _MultiProcessingDataLoaderIter(self)
  File "C:\Program Files\Python37\lib\site-packages\torch\utils\data\dataloader.py", line 719, in __init__
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Program Files\Python37\lib\multiprocessing\context.py", line 322, in _Popen
    w.start()
    return Popen(process_obj)
  File "C:\Program Files\Python37\lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__
  File "C:\Program Files\Python37\lib\multiprocessing\process.py", line 112, in start
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
    self._popen = self._Popen(self)
  File "C:\Program Files\Python37\lib\multiprocessing\context.py", line 223, in _Popen
    _check_not_importing_main()
  File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Program Files\Python37\lib\multiprocessing\context.py", line 322, in _Popen
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
    return Popen(process_obj)

I have used the exact code from given link, and if i start my code using wsl, everything is ok, but I can't use my gpu from wsl. Where should I insert this name == 'main' check to prevent such a mistake or how can i disable this multiprocessing

Upvotes: 0

Views: 1975

Answers (1)

tdelaney
tdelaney

Reputation: 77347

Looking at main.py, you run a lot of code at the module level. On Windows, python's multiprocessing module will start a new python interpreter, import your modules, unpickle a snapshot of your parent context and then call your worker function. The problem is that all of that module level code executes merely by import and you essentially run a new copy of your program instead of building a context for your worker.

The solution is two-fold. First, move all of the module level code into functions. You want to be a able to import your module without side effects. Second, call the function(s) that start your program from a conditional

def main():
    the stuff you were doing a module level

if __name__ == "__main__":
    main()

The reason this works is in the module name. When you run the top level script of a python (e.g., python main.py), its a script called "__main__", not a module. If a different program imports main its a module called "main" (or whatever you named your script). That 'if' stops your main code from executing if its imported by some other python code - such as the multiprocessing module.

Its okay to have some executable code at the module level, especially if you are setting up defaults and such. But don't do anything at the module level that you wouldn't want done if some other code imports your script.

Upvotes: 4

Related Questions