CODEWITHSUNDEEP
python-3.xclassmultiprocessingpicklepython-multiprocessing
Physicist
Physicist

Reputation: 3048

Pickling errors when using multiprocessing

I want to control and run two pieces of lab equipment at the same time (at least starts them simultaneously), so I use multiprocessing module.

from myLabModule import Newport1936R
from myLabModule import DCx_camera
from multiprocessing import Process, Queue

def multiprocess_camera(camera, shots_per_img, queue, delay):
    time.sleep(delay)
    img, camera_time = camera.capture(shots_per_img, np.float64, True) #img is a 2x2 ndarray
    results = [camera_time[0], camera_time[1], img, 'img']
    queue.put(results)

def multiprocess_power(newport, interval, N, queue, delay):
    time.sleep(delay)
    power_reading, newport_time = newport.get_power(interval, N, True)
    results = [newport_time[0], newport_time[1], power_reading, 'power']
    queue.put(results)

if __name__ == "__main__":
    thorcam = DCx_camera()
    newport = Newport1936R()
    queue = Queue()
    p1 = Process(target=multiprocess_camera, args=(thorcam,10,queue,0))
    p2 = Process(target=multiprocess_power, args=(newport,1,10,queue,0))
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    results = []
    results.append(queue.get())
    results.append(queue.get())

When I run the code, I get the following error: AttributeError: Can't pickle local object 'CDLL.__init__.<locals>._FuncPtr'. I read this post, it seems that it has to do with the scope of the variables (I am passing two classess defined in another module to the functions, and the functions basically run the classes' methods). What I'm not sure exactly which part is causing the problem. How should I modify the code to make it work?

Upvotes: 1

Views: 1657

Answers (1)

Ilija
Ilija

Reputation: 1604

In this case pickling is performed when passing arguments to target method that will be executed as new (child) process.

To be precise, I'm not sure if pickling instances of DCx_camera and Newport1936R is possible at all. Depends on how those classes are defined, but looking at the error message I suppose you can't. That is why you can try to not pass those references at all. It seems, by the example, that you do not need to.

Here is what I expect to work:

from myLabModule import Newport1936R
from myLabModule import DCx_camera
from multiprocessing import Process, Queue

def multiprocess_camera(shots_per_img, queue, delay):
    camera = DCx_camera()
    time.sleep(delay)
    img, camera_time = camera.capture(shots_per_img, np.float64, True) #img is a 2x2 ndarray
    results = [camera_time[0], camera_time[1], img, 'img']
    queue.put(results)

def multiprocess_power(interval, N, queue, delay):
    newport = Newport1936R()
    time.sleep(delay)
    power_reading, newport_time = newport.get_power(interval, N, True)
    results = [newport_time[0], newport_time[1], power_reading, 'power']
    queue.put(results)

if __name__ == "__main__":
    queue = Queue()
    p1 = Process(target=multiprocess_camera, args=(10,queue,0))
    p2 = Process(target=multiprocess_power, args=(1,10,queue,0))
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    results = []
    results.append(queue.get())
    results.append(queue.get())

Is this something that can get you desired results?

There is an alternative, according to this link.

Better to inherit than pickle/unpickle

However, one should generally avoid sending shared objects to other processes using pipes or queues. Instead you should arrange the program so that a process which needs access to a shared resource created elsewhere can inherit it from an ancestor process.

That is why I think you can even have global variables that reference instances of DCx_camera and Newport1936R, and then use those in both target methods.

Upvotes: 1

Related Questions