CODEWITHSUNDEEP
pythonpython-3.xmultithreading
Shuzheng
Shuzheng

Reputation: 13840

Is `queue.Queue` thread-safe, when accessed by several subprocesses using `concurrent.futures.ProcessPoolExecutor`?

I have been using queue.Queue extensively in situations, where I execute multiple threads e.g. by using concurrent.futures.ThreadPoolExecutor.

I've read from blogs that queue.Queue should be thread-safe, but does that mean it's thread-safe under the assumption that the Python interpreter only executes one thread at a time (GIL), or is it also thread-safe in situations using multiprocessing, which side-steps the GIL by using subprocesses instead of threads?

https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor

Upvotes: 1

Views: 1180

Answers (2)

wwii
wwii

Reputation: 23753

ProcessPoolExecutor uses multiprocessing.queues.Queue for the call queue and a mp_context.SimpleQueue (multiprocessing) for the result queue - which are used to communicate between a local thread and the processes.

Nice graphic of ProcessPoolExecutor

  • concurrent.futures.ProcessPoolExecutor stuff uses multiprocessing Queues to communicate between threads and processes.
  • The multiprocessing.queues.Queue docs specifically state it is thread and process safe
  • At the bottom of the queue documentation there is a note referring to the multiprocessing.Queue object ... for use in a multi-processing (rather than multi-threading) context

Upvotes: 2

iggy12345
iggy12345

Reputation: 1383

There is a Queue developed for this in the multiprocessing library

from multiprocessing import Queue

This uses sockets to send byte data which is thread-safe.

Upvotes: 1

Related Questions