CODEWITHSUNDEEP
pythonmultithreadingdictionary
Tim
Tim

Reputation: 3417

Is dict.update() thread-safe?

I'm wondering if dict.update() is Python thread safe. I've read the related questions, but none of them exactly addresses my question.

My question is very specific and simple. For example, I already have a local dictionary d2. I simply need to update the global dictionary d with d2 as shown below. d starts out empty and fills up with different threads. The d2 in each thread may have overlapping entries with d (don't think this matters). Is it thread safe?

import dis

def f(d):
    d2 = {1:2, 3:4}
    d.update(d2)

print(dis.dis(f))

The bytecode looks like the following:

 10           0 LOAD_CONST               1 (2)
              2 LOAD_CONST               2 (4)
              4 LOAD_CONST               3 ((1, 3))
              6 BUILD_CONST_KEY_MAP      2
              8 STORE_FAST               1 (d2)

 11          10 LOAD_FAST                0 (d)
             12 LOAD_ATTR                0 (update)
             14 LOAD_FAST                1 (d2)
             16 CALL_FUNCTION            1
             18 POP_TOP
             20 LOAD_CONST               0 (None)
             22 RETURN_VALUE

It looks like 16 CALL_FUNCTION is the atomic function that updates the dictionary. So it should be thread safe?

Upvotes: 6

Views: 2967

Answers (2)

Jab
Jab

Reputation: 27495

You could look into locked-dict if you're ok with using an external library.

From their readme:

Dict to allow context managed thread safe and mutable iterations through a lock.

For example from their tests:

pip install locked-dict

import locked_dict

expected = 0
d = locked_dict.LockedDict()
assert len(d) == expected
assert bool(d) is False
assert d is not True
assert hasattr(d, '_lock')

empty_d = {}
assert d == empty_d

plain_old_d = {999: 'plain old dict', 12345: 54321}
assert d != plain_old_d

with d as m:
    assert len(m) == expected
    assert bool(m) is False
    assert m is not True
    assert hasattr(m, '_lock')
    assert m != plain_old_d
    assert m == empty_d

    m[0] = ['foo']
    expected += 1
    assert len(m) == expected
    assert bool(m) is True
    assert m is not False
    assert m != plain_old_d
    assert m != empty_d

    m.clear()
    expected -= 1
    assert len(m) == expected
    assert bool(m) is False
    assert m is not True
    assert m != plain_old_d
    assert m == empty_d

Take not this library is 3 years old, although it may still be relevent to your use case

Upvotes: 0

Tim Peters
Tim Peters

Reputation: 70602

If the keys are compositions of builtin hashable types, generally "yes", .update() is thread-safe. In particular, for your example with integers keys, yes.

But in general, no. Looking up a key in a dict can invoke arbitrary user-defined Python code in user-supplied __hash__() and __eq__() methods, and those can do anything at all - including performing their own mutations on the dicts involved. As soon as the implementation invokes Python code, other threads can run too, including threads that may be mutating d1 and/or d2 too.

That's not a potential problem for the builtin hashable types (ints, strings, floats, tuples, ...) - their implementations to compute hash codes and decide equality are purely functional (deterministic and no side effects) and don't release the GIL (global interpreter lock).

That's all about CPython (the C implementation of Python). The answer may differ under other implementations! The Language Reference Manual is silent about this.

Upvotes: 13

Related Questions