CODEWITHSUNDEEP
pythonmultiprocessing
pow erfulfrog
pow erfulfrog

Reputation: 45

multiprocessing counter output is strange, using lock

I have a simple multiprocessing example.

I created a counter based on this, but I noticed a strange appearance during work.

from multiprocessing import Pool, Value

counter = Value('i', 0)

def init(args):
    global counter
    counter = args

def analyze_data(args):
    global counter
    with counter.get_lock():
        counter.value += 1
    print(counter.value)
    return args * 10

if __name__ == '__main__':
    counter = Value('i', 0)
    inputs = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]*10

    p = Pool(initializer = init, initargs = (counter, ),  processes=24)
    i = p.map_async(analyze_data, inputs, chunksize = 1)
    i.wait()
    print(i.get())

output

1
33
4
5
6
7
...
...
6160
6262
64
65

my expected, to 1,2,3,4,5,6...99,100 (Increase sequentially)

but, this output is strange.

Why is this happening?

Upvotes: 1

Views: 37

Answers (1)

Jean-François Fabre
Jean-François Fabre

Reputation: 140226

you have to print the counter value within the lock block to avoid a race condition between unprotected print and incrementation of the value.

with counter.get_lock():
    counter.value += 1
    print(counter.value)

else:

  • value can be changed by another process waiting on the lock before you print it (explains the double 3)
  • print statements can be executed in the wrong order (explains 60 & 61 inversion
  • the fact that 62 is printed twice is related (I must admit I don't know exactly what happens but that's the same root cause)

That or make a copy (but leaving the print inside the lock makes display nicer and avoids that outputs are mixed. That fixes the double value issue but 60 and 61 swap case can still happen):

with counter.get_lock():
    counter.value += 1
    v = counter.value
print(v)

Upvotes: 1

Related Questions