CODEWITHSUNDEEP
pythonpython-3.xmultithreadingqueue
K.Mulier
K.Mulier

Reputation: 9620

Python 3: Will Queue.empty() be correct if, for some time, no threads are putting in or pulling out elements?

I'm using a queue.Queue() in Python 3.7 (Windows 10). Sometimes I need to check if the queue is empty or not. The documentation reads:

Queue.empty()
Return True if the queue is empty, False otherwise. If empty() returns True it doesn’t guarantee that a subsequent call to put() will not block. Similarly, if empty() returns False it doesn’t guarantee that a subsequent call to get() will not block.

In other words, one cannot rely full 100% on the returned value from Queue.empty(). That seems indeed natural to me. In in a multithreaded environment, some side-thread can sneak in and put an element on the queue the very moment you're asking if it's empty. Consequently, the returned answer is unreliable. I get that.

But what about the following scenario? A timer ticks: t1, t2, t3, ... On each tick of the timer, the queue is asked: "are you empty"?

   t1       t2       t3       t4       t5       t6
---|--------|--------|--------|--------|--------|--------->
  \__________  __________/\__________  ___________/
             \/                      \/
     Different threads            No action on
     put elements on the          the queue.
     queue and/or pull
     elements from it.

Now let's imagine that from timer tick t4 onwards, we are absolutely sure that the queue isn't getting touched by any thread - no elements get inserted, nor are any pulled out.

Perhaps on timer tick t4, the queue didn't have the time yet to stabilize internally, and the returned value from Queue.empty() is wrong. But will it stabilize after some time, say on timer tick t5 or t6?

 
Possibility 1: No, it never stabilizes
That's really horrible. I will never use this queue again. End of story.

 
Possibility 2: Yes, it stabilizes after x milliseconds
I get that you cannot give an exact answer here. It will depend on many factors: the operating system, the hardware, the amount of threads that are taking resources, ... But it would just be great to know that it does stabilize after some time. I'll just take enough margin.

 
NOTE:
If Queue.empty() stabilizes (or not) after some time, I suppose the same is true for Queue.qsize(), right?

Upvotes: 2

Views: 913

Answers (1)

Martijn Pieters
Martijn Pieters

Reputation: 1121794

It will stabilize. When no threads interact with the queue there is no possibility for a race condition any more.

The Queue.empty() method returns True or False based on the current state of the queue. At issue is that that state can change at any moment when another thread is given control and interacts with the queue, so within one thread, barring other synchronisation primitives (locks, semaphores, events, conditions, barriers, what-have-you), you can't rely on the return value for any length of time.

There is therefor no timeframe in which you can count on the state stabilising. There is no 'unstable state' in the queue object, only in your application as a whole, because you are using threads.

The same applies to the Queue.qsize() method; the return value is not fuzzy or approximate, it is an accurate measure of the state of the queue at that moment in time. But since your code can and will be interrupted when control is transferred to another thread, you can't rely on that measurement to inform how your code can then act on the queue, because by the time the next instruction in your thread could act on that measurement, control may have been switched away and back again, another thread could have acted on the queue, and the measurement is no longer relevant.

Use other synchronisation primitives to communicate between your threads if you need something more definitive.

Upvotes: 4

Related Questions