CODEWITHSUNDEEP
python-2.7multiprocessing
Vanshika
Vanshika

Reputation: 45

Multiprocessing with Python 2.7 throwing attribute error

from itertools import product
from multiprocessing import Pool

with Pool(4) as pool:
            pool.map(lambda x: run_test_function(x, arg2, arg3, arg4), arg1)

I am getting below error after executing above code. There some other code as well which I can't write here. But actual problem is coming from this piece of code only.

Traceback (most recent call last):
  File "modProfileChange_test.py", line 347, in <module>
    main(sys.argv[1:])
  File "modProfileChange_test.py", line 336, in main
    test_run_code(arg1, arg2, arg3, arg4, arg5, arg6)
  File "modProfileChange_test.py", line 23, in test_run_code
    with Pool(4) as pool:
AttributeError: __exit__

Upvotes: 0

Views: 167

Answers (1)

ForceBru
ForceBru

Reputation: 44838

  1. In Python 2.7, multiprocessing.Pool is not a context manager and thus it can't be used in a with statement
    1. Solution - create a pool using regular assignment to a variable: 
      my_pool = Pool(4)
my_pool.map(...)
  2. lambda functions don't work with multiprocessing.Pool, even in Python 3.
    1. Solution - emulate a closure using a solution in the link above: 
      from functors import partial

def run_test_function(x, fun_arg2, fun_arg3, fun_arg4):
    # your code here
    ...

process_func = partial(run_test_function, fun_arg2=arg2, fun_arg3=arg3, fun_arg4=arg4)

Putting this together:

from multiprocessing import Pool
from functools import partial

def run_test_function(x, fun_arg2, fun_arg3, fun_arg4):
    # this is an example
    print x, fun_arg2, fun_arg3, fun_arg4

if __name__ == "__main__":
    arg1 = 1,2,3,4
    arg2 = "hello"
    arg3 = "world"
    arg4 = "!"

    process_func = partial(run_test_function, fun_arg2=arg2, fun_arg3=arg3, fun_arg4=arg4)

    my_pool = Pool(4)
    my_pool.map(process_func, arg1)

Output:

~/test $ python2.7 so10.py
1 hello world !
2 hello world !
3 hello world !
4 hello world !

Upvotes: 2

Related Questions