CODEWITHSUNDEEP
pythonmultithreading
DNS
DNS

Reputation: 38219

Can one iterate a dictionary being modified by another thread without raising exceptions?

I have a dictionary being updated by one thread, and in another thread I'd like to iterate over its values. Normally I'd use a lock, but this code is very performance-critical, and I want to avoid that if at all possible.

A special feature of my case is that I don't care about absolute correctness of the iterator; if it misses entries that were removed after iteration started, or picks up ones added afterwards, that's fine. I only require that it doesn't raise any sort of 'dictionary size changed during iteration' exception.

Given this relaxed constraint on correctness, is there an efficient way to iterate the dictionary without using a lock?

Note: I'm aware that keys() is threadsafe in Python 2.x, but since that behavior has changed in 3.x, I want to avoid it.

Upvotes: 6

Views: 1896

Answers (4)

don_vanchos
don_vanchos

Reputation: 1540

Sometimes an example is better than words.

Array iteration is NOT thread-safe, see live example for Python 3.6

Upvotes: 1

Matt Anderson
Matt Anderson

Reputation: 19809

I would consider using a lock long enough to retrieve the values that you want to iterate over:

with lock:
    values = the_dict.values() # python 2
    # values = list(the_dict.values()) # python 3
for value in values:
    # do stuff

Or, you could try it without a lock and catch RuntimeError, and if you get one, try to retrieve the values again.

[edit] Below slightly rephrased per J.F. Sebastian's suggesion:

while True:
    try:
        values = list(the_dict.values())
        break
    except RuntimeError:
        pass

I personally would go with the lock.

Upvotes: 3

Rhand
Rhand

Reputation: 897

No personal experience with this, but I read this some time ago: http://www.python.org/dev/peps/pep-3106/

These operations are thread-safe only to the extent that using them in a thread-unsafe way may cause an exception but will not cause corruption of the internal representation.

As in Python 2.x, mutating a dict while iterating over it using an iterator has an undefined effect and will in most cases raise a RuntimeError exception. (This is similar to the guarantees made by the Java Collections Framework.)

Upvotes: 4

Ali Afshar
Ali Afshar

Reputation: 41667

Two things:

  1. Dump the keys into a queue and read that safely.
  2. Performance-critical code should probably not be using Python threads.

Upvotes: 2

Related Questions