CODEWITHSUNDEEP
pythonpython-2.7python-multiprocessing
MuertoExcobito
MuertoExcobito

Reputation: 10039

multiprocessing and modules

I am attempting to use multiprocessing to call derived class member function defined in a different module. There seem to be several questions dealing with calling class methods from the same module, but none from different modules. For example, if I have the following structure:

main.py
multi/
    __init__.py (empty)
    base.py
    derived.py

main.py

from multi.derived import derived
from multi.base import base

if __name__ == '__main__':
    base().multiFunction()
    derived().multiFunction()

base.py

import multiprocessing;

# The following two functions wrap calling a class method
def wrapPoolMapArgs(classInstance, functionName, argumentLists):
    className = classInstance.__class__.__name__
    return zip([className] * len(argumentLists), [functionName] * len(argumentLists), [classInstance] * len(argumentLists), argumentLists)

def executeWrappedPoolMap(args, **kwargs):
    classType = eval(args[0])
    funcType = getattr(classType, args[1])
    funcType(args[2], args[3:], **kwargs)

class base:
    def multiFunction(self):
        mppool = multiprocessing.Pool()
        mppool.map(executeWrappedPoolMap, wrapPoolMapArgs(self, 'method', range(3)))

    def method(self,args):
        print "base.method: " + args.__str__()

derived.py

from base import base

class derived(base):
    def method(self,args):
        print "derived.method: " + args.__str__()

Output

base.method: (0,)
base.method: (1,)
base.method: (2,)
Traceback (most recent call last):
  File "e:\temp\main.py", line 6, in <module>
    derived().multiFunction()
  File "e:\temp\multi\base.py", line 15, in multiFunction
    mppool.map(executeWrappedPoolMap, wrapPoolMapArgs(self, 'method', range(3)))
  File "C:\Program Files\Python27\lib\multiprocessing\pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "C:\Program Files\Python27\lib\multiprocessing\pool.py", line 567, in get
    raise self._value
NameError: name 'derived' is not defined

I have tried fully qualifying the class name in the wrapPoolMethodArgs method, but that just gives the same error, saying multi is not defined.

Is there someway to achieve this, or must I restructure to have all classes in the same package if I want to use multiprocessing with inheritance?

Upvotes: 1

Views: 144

Answers (1)

ShadowRanger
ShadowRanger

Reputation: 155363

This is almost certainly caused by the ridiculous eval based approach to dynamically invoking specific code.

In executeWrappedPoolMap (in base.py), you convert a str name of a class to the class itself with classType = eval(args[0]). But eval is executed in the scope of executeWrappedPoolMap, which is in base.py, and can't find derived (because the name doesn't exist in base.py).

Stop passing the name, and pass the class object itself, passing classInstance.__class__ instead of classInstance.__class__.__name__; multiprocessing will pickle it for you, and you can use it directly on the other end, instead of using eval (which is nearly always wrong; it's code smell of the strongest sort).

BTW, the reason the traceback isn't super helpful is that the exception is raised in the worker, caught, pickle-ed, and sent back to the main process and re-raise-ed. The traceback you see is from that re-raise, not where the NameError actually occurred (which was in the eval line).

Upvotes: 1

Related Questions