CODEWITHSUNDEEP
pythonthread-safetydeque
user1102018
user1102018

Reputation: 4451

Is this deque thread-safe in python?

I can't decide whether the following deque is thread-safe.
In short, I've created a class with a deque that displays its contents every 1 sec in a new thread (so it won't pause the main program while printing).
The deque is filled from the main thread, so basically there SHOULD be a chance of collision.
HOWEVER, the deque is filled using a class method, so essentially it is accessed from within the instance itself, therefore from the same thread.
Here's the simplified code:

import threading
import time
from collections import deque

class MyQueue(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)
        self.q = deque()
        self.start()

    def run(self):
        # pop out queue items every 1 sec
        # (please ignore empty deque for now)
        while True:
            print self.q.popleft()
            time.sleep(1)

    def add_to_q(self, val):
        # this function is called from outside
        self.q.append(val)

# main
# fill the queue with values
qu = MyQueue()
for i in range(1:100):
    qu.add_to_q(i)

So, although adding and removing items from queue take place inside the instance, is there a risk due to the adding function being called from outside the instance?

EDIT:
Since I need to modify items in my deque, I had to use Deque. What I do is: roatate() to the given item, pop it out, modify, push it back in and rotate() it back to its original position.
Unless I find a way of implementing modifying items in a Queue, I'll have to stick to Deque

Upvotes: 29

Views: 37260

Answers (3)

Raymond Hettinger
Raymond Hettinger

Reputation: 226296

Deque author here.

The MyQueue() class looks correct to me (at least with respect to the question of thread-safety).

The append() and popleft() methods are both atomic.

The code does need EAFP logic to handle the case where the input is empty:

def run(self):
    while True:
        try:
            print self.q.popleft()
        except IndexError:
            pass
        time.sleep(1)

Upvotes: 31

BadWolf
BadWolf

Reputation: 375

For information there is a Python ticket referenced for deque thread-safety (https://bugs.python.org/issue15329).

Title "clarify which deque methods are thread-safe", bottom line is:

The deque's append(), appendleft(), pop(), popleft(), and len(d) operations are thread-safe in CPython. The append methods have a DECREF at the end (for cases where maxlen has been set), but this happens after all of the structure updates have been made and the invariants have been restored, so it is okay to treat these operations as atomic.

Anyway, if you are not 100% sure and you prefer reliability over performance, just put a like Lock for print self.q.popleft() and self.q.append(val) ;)

Upvotes: 11

King
King

Reputation: 1200

Deque is thread-safe (http://docs.python.org/library/collections.html#deque-objects) for appends and pops from opposite sides. Beneath here, the docs only mention that append() and popleft() are thread-safe.

There is a thread-safe implementation of the Queue itself. So you should be using it unless you have some strange requirements.

Upvotes: 32

Related Questions