CODEWITHSUNDEEP
pythonmultithreadingvariablesdictionarylocking
Jelle De Loecker
Jelle De Loecker

Reputation: 21985

Modifying a Python dictionary from different threads

When it comes to threading, I know you have to make sure you're not editing a variable at the same time another thread is editing it, as your changes can be lost (when incrementing a counter, for example)

Does the same apply to dictionaries? Or is a dictionary a collection of variables?

If every thread were to lock the dictionary it would slow the program down significantly, while every thread only needs write access to its own little piece of the dictionary.

If it isn't possible, is there some sort of variable variable in python, like in php?

Upvotes: 16

Views: 17246

Answers (3)

mouad
mouad

Reputation: 70079

Does the same apply to dictionaries? Or is a dictionary a collection of variables?

Let's be more general:

What does "atomic operation" mean?

From Wikipedia :

In concurrent programming, an operation (or set of operations) is atomic, linearizable, indivisible or uninterruptible if it appears to the rest of the system to occur instantaneously. Atomicity is a guarantee of isolation from concurrent processes.

Now what does this mean in Python?

This means that each bytecode instruction is atomic (at least for Python <3.2, before the new GIL).

Why is that???

Because Python (CPython) use a Global Interpreter Lock (GIL). The CPython interpreter uses a lock to make sure that only one thread runs in the interpreter at a time, and uses a "check interval" (see sys.getcheckinterval()) to know how many bytecode instructions to execute before switching between threads (by default set to 100).

So now what does this mean??

It means that operations that can be represented by only one bytecode instruction are atomic. For example, incrementing a variable is not atomic, because the operation is done in three bytecode instructions:

>>> import dis

>>> def f(a):
        a += 1

>>> dis.dis(f)
  2           0 LOAD_FAST                0 (a)
              3 LOAD_CONST               1 (1)      <<<<<<<<<<<< Operation 1 Load
              6 INPLACE_ADD                         <<<<<<<<<<<< Operation 2 iadd
              7 STORE_FAST               0 (a)      <<<<<<<<<<<< Operation 3 store
             10 LOAD_CONST               0 (None)
             13 RETURN_VALUE

So what about dictionaries??

Some operations are atomic; for example, this operation is atomic:

d[x] = y
d.update(d2)
d.keys()

See for yourself:

>>> def f(d):
        x = 1
        y = 1
        d[x] = y

>>> dis.dis(f)
  2           0 LOAD_CONST               1 (1)
              3 STORE_FAST               1 (x)

  3           6 LOAD_CONST               1 (1)
              9 STORE_FAST               2 (y)

  4          12 LOAD_FAST                2 (y)
             15 LOAD_FAST                0 (d)
             18 LOAD_FAST                1 (x)
             21 STORE_SUBSCR                      <<<<<<<<<<< One operation 
             22 LOAD_CONST               0 (None)
             25 RETURN_VALUE

See this to understand what STORE_SUBSCR does.

But as you see, it is not totally true, because this operation:

             ...
  4          12 LOAD_FAST                2 (y)
             15 LOAD_FAST                0 (d)
             18 LOAD_FAST                1 (x)
             ...

can make the entire operation not atomic. Why? Let's say the variable x can also be changed by another thread...or that you want another thread to clear your dictionary...we can name many cases when it can go wrong, so it is complicated! And so here we will apply Murphy's Law: "Anything that can go wrong, will go wrong".

So what now?

If you still want to share variables between thread, use a lock:

import threading

mylock = threading.RLock()

def atomic_operation():
    with mylock:
        print "operation are now atomic"

Upvotes: 26

Apalala
Apalala

Reputation: 9244

What you need to do is to not allow the threads direct access to the shared data structure but instead wrap access to it with something that guarantees mutual exclusion like a mutex.

Making access to the original structure look the same (shared[id] = value) takes some more work, but not that much.

Upvotes: 0

user395760
user395760

Reputation:

I think you misundertood this whole thread safety thing. It's not so much about variables (or variable variables - those are terrible anyway, and are just as pointless - not to say harmful - here as in every other case) but about -- for example, there are many nasty nasty ways threading can go wrong; they all come from accessing something mutable from more than one thread at overlapping times -- this:

  • thread N gets data from source (some place in memory or on disk - a variable, a slot in a dictionary, a file, pretty much anything mutable)
  • thread M gets data from source
  • thread N modifies the data
  • thread M modifies the data
  • thread N overwrites source with modified data
  • thread M overwrites source with modified data
  • Result: thread N's modifications are lost/the new shared value doesn't take thread N's modifications into account

And it applies to dictionaries and variable variables (which are just a horrible, horrible language-level implementation of dicts with string-only keys) as well. The only solutions are not using shared state to begin with (functional languages do this by discouraging or even completely disallowing mutability, and it works well for them) or adding some sort of locking to everything shared (hard to get right, but if you get it right, at least it works correctly). If no two threads every share anything in that dictionary, you're fine - but you should seperate everything, to be (a bit more) sure that they really don't share anything.

Upvotes: 8

Related Questions