CODEWITHSUNDEEP
pythonmultithreadingpython-multithreading
Sriram
Sriram

Reputation: 1728

Python Lock and GIL in Multithreading

If Python has GIL and even multi threaded programs doesn't run concurrently unless they are I/O bound, do we really need Lock concept in Python ?

Upvotes: 1

Views: 366

Answers (1)

abarnert
abarnert

Reputation: 365767

The simple answer is that you need locks around each operation on shared mutable data, and what “an operation” means to your algorithm can be much larger than what the GIL protects.

It’s often easier to understand things with a concrete example than with abstractions, so let’s come up with one. You have an iterable of lines, and you want to count the words. For each line, you call this function:

def updatecounts(counts, line):
    for word in line.split():
        if word in counts:
            counts[word] += 1
        else:
             counts[word] = 1

Now, instead of just calling updatecounts in a loop, you create a thread executor or pool and call pool.map(partial(updatecounts, count), lines). (OK, that would be silly, but say you had 100 client sockets producing lines; then it would make sense to have threads that called this function in the middle of their other work.)

Thread 1 is working on line 1, which starts with “Now”. It checks whether ”Now” is in counts. It isn’t. So… but then the thread is interrupted and thread 3 takes over. Its line also starts with “Now”, so it checks whether ”Now” is in counts. Nope, so it sets counts["Now"] to 1. Then it moves on to the next word and… at some point, thread 1 starts executing again. And what was it about to do? It sets counts["Now"] to 1. And we’ve just lost a count.

The way to prevent this is to pass a lock around:

def updatecounts(counts, countslock, line): for word in line.split(): with countslock: if word in counts: counts[word] += 1 else: counts[word] = 1

Now, if thread 1 gets interrupted after checking if word in counts:, it’s still holding countslock. So when thread 3 tries to acquire the same countslock, it can’t; it blocks until the lock is free. The system might run some other threads for a while, but ultimately it’s guaranteed to come back to thread 1 so it can finish its work and release the lock, before thread 3 can do anything.

Why doesn’t the GIL protect us here? Because the GIL has no idea that you want that whole four lines protected.

What if we just used a Counter, so we could write counts[word] += 1? Well, that may only be one line of source code, but it still compiles to multiple bytecodes, and the level the GIL actually protects is bytecodes.

In fact, it’s not at all obvious what a “bytecode” is from the point of view of your code. You can work it out with the help of the dis module, but even then it’s not always clear. For example, that words in count is compiled to a single bytecode that does the comparison—except that bytecode actually calls the __contains__ method on counts. CPython implements dict.__contains__ in C rather than Python, and doesn’t release the GIL. But what if counts might be some mapping implemented in Python (like a Counter) that takes multiple bytecodes to implement the method? Or, even with a dict, __contains__(word) has to ultimately call word.__hash__. Can that release the GIL?

Occasionally, when you really need to optimize some inner loop, it’s worth doing all the work the verify that counts is definitely a dict and word is definitely a str and all of the operations are guaranteed in the docs (or, if not there, by reading the C source code) to hold the GIL, and therefore you can be sure that word in counts is atomic.

Well, you can be sure it is in CPython 3.7; if your code has to run on 3.5 or 2.7 you have to check there as well. And if it has to run on Jython, Jython doesn’t even have a GIL…

Plus, it’s rare that you need to micro-optimize code inside of a threaded inner loop in the first place, because that implies that your code is CPU-bound, in which case you probably shouldn’t have been using threads and shared variables in the first place.

Upvotes: 1

Related Questions