CODEWITHSUNDEEP
pythonmultithreading
L.fole
L.fole

Reputation: 807

how to make a thread-safe global counter in python

I'm creating a threading.Timer(2,work) run threads. Inside each work function, upon some condition the global counter must increment without conflict for access of counter variable among the spawned work threads.

I've tried Queue.Queue assigned counter as well as threading.Lock(). Which is a best way to implement thread-safe global increment variable.

Previously someone asked question here: Python threading. How do I lock a thread?

Upvotes: 33

Views: 46931

Answers (2)

PhilMarsh
PhilMarsh

Reputation: 44

If you're using CPython1, you can do this without explicit locks:

import itertools

class Counter:
    def __init__(self):
        self._incs = itertools.count()
        self._accesses = itertools.count()

    def increment(self):
        next(self._incs)

    def value(self):
        return next(self._incs) - next(self._accesses)

my_global_counter = Counter()

We need two counters: one to count increments and one to count accesses of value(). This is because itertools.count does not provide a way to access the current value, only the next value. So we need to "undo" the increments we incur just by asking for the value.

This is threadsafe because itertools.count.__next__() is atomic in CPython (thanks, GIL!) and we don't persist the difference.

Note that if value() is accessed in parallel, the exact number may not be perfectly stable or strictly monotonically increasing. It could be plus or minus a margin proportional to the number of threads accessing. In theory, self._incs could be updated first in one thread while self._accesses is updated first in another thread. But overall the system will never lose any data due to unguarded writes; it will always settle to the correct value.

1 Not all Python is CPython, but a lot (most?) is.

2 Credit to https://julien.danjou.info/atomic-lock-free-counters-in-python/ for the initial idea to use itertools.count to increment and a second access counter to correct. They stopped just short of removing all locks.

Upvotes: 1

mommermi
mommermi

Reputation: 1052

Not sure if you have tried this specific syntax already, but for me this has always worked well:

Define a global lock:

import threading
threadLock = threading.Lock()

and then you have to acquire and release the lock every time you increase your counter in your individual threads:

with threadLock:
    global_counter += 1

Upvotes: 55

Related Questions