CODEWITHSUNDEEP
python-3.x
Larissa
Larissa

Reputation: 11

python multiprocessing.pool() OSError: [Errno 22] Invalid argument

I try to run this code:

import numpy as np

import multiprocessing
from functools import partial

CPUS = multiprocessing.cpu_count() # Anzahl der CPUs/Threads: 4
anzahl = CPUS-1

WORKERS = multiprocessing.Pool(processes=anzahl)

I tried to run it on 3 different computers: on 2 all works fine, but on 1 computer I get the following traceback error:

Traceback (most recent call last):

  File "<string>", line 1, in <module>

  File "L:\Python\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)

  File "L:\Python\lib\multiprocessing\spawn.py", line 125, in _main
    prepare(preparation_data)

  File "L:\Python\lib\multiprocessing\spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])

  File "L:\Python\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,

  File "L:\Python\lib\runpy.py", line 264, in run_path
    code, fname = _get_code_from_file(run_name, path_name)

  File "L:\Python\lib\runpy.py", line 234, in _get_code_from_file
    with io.open_code(decoded_path) as f:

OSError: [Errno 22] Invalid argument: 'L:\\Python\\<input>'

I don't know how to solve this problem. Can someone help me? My Python version: Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)] on win32

Upvotes: 1

Views: 2829

Answers (3)

DEEPAK S.V.
DEEPAK S.V.

Reputation: 95

Running the Pool function inside of the if __name__ == '__main__': block solved the issue for me.

For example,

from multiprocessing import Pool

def mul2(x):
    return x*2

if __name__ == '__main__':
    p = Pool(4)

    results = p.map(mul2, range(4))
    p.close()
    p.join()

    print(results)

Upvotes: 0

Squeaky Boots
Squeaky Boots

Reputation: 1

Credit to those in the comments of how-to-create-multiprocessingpool-object

import collections
import multiprocessing
import time
from pprint import pprint

# this needs to be outside of if name == main or you get a pickle error
Scientist = collections.namedtuple('Scientist', ['name', 'field', 'born', 'nobel'])

# this might be ok inside if name == main but it might as well go here
scientists = (Scientist('Ada Lovelace', 'math', 1815, False), Scientist('Emmy Noether', 'math', 1882, False),
              Scientist('Marie Curie', 'math', 1867, True), Scientist('Tu Youyou', 'physics', 1930, True),
              Scientist('Ada Yonath', 'chemistry', 1939, True), Scientist('Vera Rubin', 'astronomy', 1928, False),
              Scientist('Sally Ride', 'physics', 1951, False))


def transform(x):
    print(f'Processing record {x.name}')
    time.sleep(1)
    result = {'name': x.name, 'age': 2021 - x.born}
    print(f'Done processing record {x.name}')
    return result


def run():
    start = time.time()

    pool = multiprocessing.Pool()
    result = pool.map(transform, scientists)

    # result = tuple(map(
    #     transform,
    #     scientists
    # ))

    end = time.time()

    print(f'\nTime to complete: {end - start:2f}s\n')
    pprint(result)


if __name__ == '__main__':
    run()
    # multiprocessing.freeze_support()
    # multiprocessing.set_start_method("spawn")

This thread talks about it as well: Can multiprocessing Process class be run from IDLE

Upvotes: 0

GydeonMylls
GydeonMylls

Reputation: 21

A workaround to this (tested in PyCharm 2020.3.2) is to change the __file__ console variable to the name of an existing file before instantiating the Pool

Upvotes: 2

Related Questions