CODEWITHSUNDEEP
pythonserializationpython-multiprocessingdillpathos
Shubham Kedia
Shubham Kedia

Reputation: 77

Multiprocessing in Python using Pool Class

I want to use multiprocessing in python(specifically Pool Class) in windows(zeppelin notebook). But after reading a lot I got to know that the Pool Class uses pickle in its back end and can be use for only the objects that can be pickled. As I can't serialize a function or a class using pickle in Python(at least in 3.6 version), I found out that pathos.multiprocessing library can be used to perform multiprocessing as it uses dill instead of pickle in the back end which according to dill package documentation can serialize anything. But when I used it I got errors. Below is the simple code that I used:

import pathos.multiprocessing as mp

def cube(x):
    return x**3

p = mp.Pool(4)

p.apply(cube, [1,2,3])

And this is the error I got:

py4j.protocol.Py4JError: An error occurred while calling o0.__getstate__. Trace:
py4j.Py4JException: Method __getstate__([]) does not exist
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:335)
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:344)
    at py4j.Gateway.invoke(Gateway.java:279)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:209)
    at java.lang.Thread.run(Thread.java:748)

I got the same error even if I used map, apply_async or map_async. Can someone help me finding out the issue.

Thanks in advance

Upvotes: 0

Views: 345

Answers (1)

Shubham Kedia
Shubham Kedia

Reputation: 77

Also If i use the code like the this:

import multiprocessing as mp

def cube(x):
    return x**3

if __name__ == '__main__':
    p = mp.Pool(4)
    p.apply(cube, [1,2,3])

I don't see any output returned. Not sure what the issue is.

Upvotes: 0

Related Questions