CODEWITHSUNDEEP
pythonmultiprocessingpickle
Alina Belova
Alina Belova

Reputation: 13

Error when trying to use pickle and multiprocessing

I'm trying to use multiprocessing to run my code in Python 3.7, but met a problem. There is an error when I try to run my code:

Can't pickle local object 'mm_prepare_run.<locals>...

I understand it's a problem with pickle, but I didn't find a proper answer how to resolve this issue.

My simple code is below. Could you advise how I can solve the problem?

import multiprocessing
import copy
from pathlib import Path

proc_mrg = multiprocessing.Manager()
num_cpu = 8   # number of CPU


def prepare_run(config):
    din['config'] = config
    din_temp = copy.deepcopy(din)
    dout_list.append(proc_mrg.dict({}))

    #process = multiprocessing.Process(target=Run_IDEAS_instance_get_trajectory,args=(din_temp, dout_list[-1]))  
    process = multiprocessing.Process(target=Run_IDEAS_instance_get_trajectory(din_temp, dout_list[-1]))
    proc_list.append(process)

    for job in proc_list:
        job.start()

Upvotes: 1

Views: 228

Answers (1)

Roland Smith
Roland Smith

Reputation: 43495

When you create a Process, in prepare_run you are calling Run_IDEAS_instance_get_trajectory instead of just passing it as a reference. And since that function does not return a result, the target of the Process is None.

Use this instead:

process = multiprocessing.Process(
    target=Run_IDEAS_instance_get_trajectory,
    args=(din_temp, dout_list[-1])
)

Functions in Python are first class objects of the callable type. See the "Data model" chapter in the Python language reference.

Edit:

From your comment, I can see that you are running this code on ms-windows. On this platform it is required that you run process creation inside a if __name__ == "__main__" block! Because of how multiprocessing works on this platform, python has to be able to import your script without side effects such as starting a new process. See the "programming guidelines" section in the documentation for multiprocessing.

Upvotes: 1

Related Questions