CODEWITHSUNDEEP
pythonpython-3.xpython-multiprocessing
davidbak
davidbak

Reputation: 5999

Why does multiprocessing.Queue have a small delay while (apparently) multiprocessing.Pipe does not?

Documentation for multiprocessing.Queue points out that there's a bit of a delay from when an item is enqueued until it's pickled representation is flushed to the underlying Pipe. Apparently though you can enqueue an item straight into a Pipe (it doesn't say otherwise and implies that's the case).

Why doesn't the Pipe need - or have - the same background thread to do the pickling? And is that the same reason that there's no similar delay when talking to a multiprocessor.SyncManager.Queue?

(Bonus question: What does the documentation mean when it says "After putting an object on an empty queue there may be an infinitesimal delay ..."? I've taken calculus; I know what infinitesimal means, and that meaning doesn't seem to fit here. So what is it talking about?)

Upvotes: 1

Views: 761

Answers (1)

Davis Herring
Davis Herring

Reputation: 39808

If you write to a Pipe, the current thread blocks until the write completes. There is therefore no delay (or rather, the calling thread can’t observe any), but it’s possible to deadlock; Pipe is a lower-level tool than Queue.

The situation with SyncManager.Queue is simply that all requests to the manager are synchronized such that the process that pushes an object cannot then observe it to still be empty (absent a pop).

Meanwhile, the “infinitesimal” delay merely means a thread scheduling delay rather than the (possibly much larger) time to write the entire object: it’s enough for it to have started so as to establish that the Queue is not empty. The pushing thread can nonetheless win the race and observe it still lacking the object “already pushed”.

Upvotes: 1

Related Questions