CODEWITHSUNDEEP
pythonmultiprocessingpickledillpathos
Ryan
Ryan

Reputation: 80

Does pathos only work utilizing a pool? I'm looking to get around a pickle issue with multiprocessing.queues

I'm trying to create a series of worker threads that have specific jobs. The current is to draw from a queue, do some processing, and put the output in a second queue.

When I try to use the multiprocessing (multiprocess? what's the difference?) module I get the following issue

Traceback (most recent call last):
  File "C:\Python27\lib\multiprocessing\queues.py", line 264, in _feed
    send(obj)
PicklingError: Can't pickle <type 'function'>: attribute lookup     __builtin__.function failed

It seems to be a cpickle issue of some kind. I am processing objects with __getattr__ overwritten but I did use

if name.startswith('__') and name.endswith('__'):
        return object.__getattr__(name)

to ensure that the TypeError is not called anymore. The answer to pickling issues with multiprocessing is to use pathos but I don't see a way to access any kind of queues in pathos. I also don't see a way to spawn a Process in pathos. Every example I've seen uses a pool. I also saw a response that there is pathos.helpers.mp.process.Process but the modules cannot be found for me - ImportError: cannot import name helpers, for example. Is this something possible?

Upvotes: 0

Views: 539

Answers (1)

Mike McKerns
Mike McKerns

Reputation: 35247

I'm the pathos author. Yes, you can use a Process from pathos, however, if that is what you are interested in, it's probably easier to just use multiprocess. Some disambiguation: multiprocess is a fork of multiprocessing, where the fork replaces pickle with dill -- and there's no other changes. pathos, then adds an additional layer on top of multiprocess, primarily on the Pool objects. pathos.multiprocessing is the additional layer on top of multiprocess… and if you want direct access to multiprocess from pathos, you can get it. See:

>>> import pathos
>>> pathos.helpers.mp.Process
<class 'multiprocess.process.Process'>
>>> 
>>> import multiprocess as mp
>>> mp.Process
<class 'multiprocess.process.Process'>
>>>

Upvotes: 1

Related Questions