CODEWITHSUNDEEP
pythonmultiprocessing
user3599803
user3599803

Reputation: 7014

multiprocessing initargs - how it works under the hood?

  1. I've assumed that multiprocessing.Pool uses pickle to pass initargs to child processes.

However I find the following stange:

value = multiprocessing.Value('i', 1)
multiprocess.Pool(initializer=worker, initargs=(value, )) # Works

But this does not work:
pickle.dumps(value)
throwing:

RuntimeError: Synchronized objects should only be shared between processes through inheritance

Why is that, and how multiprocessing initargs can bypass that, as it's using pickle as well?

  1. As I understand, multiprocessing.Value is using shared memory behind the scenes, what is the difference between inheritance or passing it via initargs? Specifically speaking on Windows, where the code does not fork, so a new instance of multiprocessing.Value is created.

Upvotes: 1

Views: 499

Answers (1)

Booboo
Booboo

Reputation: 44148

And if you had instead passed an instance of multiprocessing.Lock(), the error message would have been RuntimeError: Lock objects should only be shared between processes through inheritance. These things can be passed as arguments if you are creating an instance of multiprocessing.Process, which is in fact what is being used when you say initializer=worker, initargs=(value,). The test being made is whether a process is currently being spawned, which is not the case when you already have an existing process pool and are now just submitting some work for it. But why this restriction?

Would it make sense for you to be able to pickle this shared memory to a file and then a week later trying to unpickle it and use it? Of course not! Python cannot know that you would not be doing anything so foolish as that and so it places great restrictions on how shared memory and locks can be pickled/unpickled, which is only for passing to other processes.

Upvotes: 1

Related Questions