CODEWITHSUNDEEP
pythonmultithreadingmultiprocessingsemaphore
Danegraphics
Danegraphics

Reputation: 477

Python Multiprocessing Semaphore not working

I want my program to print out one line at a time, however it's printing multiple at a time and creating a garbled mess. I cannot seem to find out why the semaphore is not preventing multiple processes from printing over each other.

How can I get it to respect the semaphore?

Here is a simplified version of my code that has the same problem when run it (I'm running on Windows with Python 2.7.11 (this can't be changed)):

import multiprocessing

rightofway = multiprocessing.Semaphore(1)

def writenum(number):
    rightofway.acquire()
    print("[+] - " + str(number))
    rightofway.release()
    return

def main():
    starting = 0
    ending = 50

    list = range(starting, ending)

    pool = multiprocessing.Pool(10)
    pool.map(writenum, list)
    return

#Required for Windows multiprocessing
if __name__ == '__main__':
    main()

Here is an example of garbled output:

[+] - 0
[+] - 1
[+] - 2
[+] - 3
[+] - 4
[+] - 5
[+] - 6
[+] - 7
[[+] - 8
+] - 10[
+] - 9[+] - 11
[+] - 12

[[+] - 13+] - 14

[[+] - 15+] - 16

[[+] - 18+] - 17

[[+] - 19+] - 20

[[+] - 22+] - 21

[[+] - 23+] - 24

[[+] - 26+] - 25

[[+] - 27+] - 28

[[+] - 30+] - 29

[[+] - 31+] - 32

[[+] - 34+] - 33

[[+] - 35+] - 36

[[+] - 38+] - 37

[[+] - 39+] - 40

[[+] - 42+] - 41

[[+] - 43+] - 44

[[+] - 46+] - 45

[[+] - 47+] - 48

[+] - 49

Here's an example of the output I want (note I don't care about the order):

[+] - 0
[+] - 1
[+] - 2
[+] - 3
[+] - 4
[+] - 5
[+] - 6
[+] - 7
[+] - 8
[+] - 9
[+] - 10
[+] - 11
[+] - 12
[+] - 13
[+] - 14
[+] - 15
[+] - 16
[+] - 17
[+] - 18
[+] - 19
[+] - 20
[+] - 21
[+] - 22
[+] - 23
[+] - 24
[+] - 25
[+] - 26
[+] - 27
[+] - 28
[+] - 29
[+] - 30
[+] - 31
[+] - 32
[+] - 33
[+] - 36
[+] - 34
[+] - 35
[+] - 37
[+] - 38
[+] - 40
[+] - 39
[+] - 41
[+] - 42
[+] - 44
[+] - 43
[+] - 45
[+] - 46
[+] - 48
[+] - 47
[+] - 49

Upvotes: 5

Views: 6485

Answers (2)

noxdafox
noxdafox

Reputation: 15060

You question is similar to this one.

From the multiprocessing programming guidelines.

Explicitly pass resources to child processes

... it is better to pass the object as an argument to the constructor for the child process.

Apart from making the code (potentially) compatible with Windows ...

On Windows, you need to pass the shared objects to the Process constructor list of arguments. Otherwise, the child process will get a brand new copy instead of the parent's one. That's why you get the impression the Semaphore isn't working. The two processes are creating their own distinct Semaphore object instead of sharing the same one.

To pass a Semaphore object to a Pool on Windows you need to struggle a bit but not too much. As you cannot pass the Semaphore object to the writenum function directly, you need to rely on the Pool initializer.

from multiprocessing import Semaphore, Pool

mutex = None

def initializer(semaphore):
    """This function is run at the Pool startup. 
    Use it to set your Semaphore object in the child process.

    """
    global mutex

    mutex = semaphore

def writenum(args):
    with mutex:
        print "[+] - " + str(number)

def main():
    semaphore = Semaphore()
    pool = Pool(initializer=initializer, initargs=[semaphore])

    numbers = range(50)

    pool.map(writenum, numbers)

EDIT: just noticed I wrote about Lock instead of Semaphore. The core reasoning remains the same.

Upvotes: 5

Ishan Bhatt
Ishan Bhatt

Reputation: 10269

To make things a bit easier, Following worked for me. Tested on Win10. TL;DR - Use locks not semaphore

import multiprocessing

rightofway = multiprocessing.Lock()

def writenum(number):

    with rightofway:
        print("[+] - " + str(number))

    return

def main():
    starting = 0
    ending = 50

    list = range(starting, ending)

    pool = multiprocessing.Pool(10)
    pool.map(writenum, list)
    return

#Required for Windows multiprocessing
if __name__ == '__main__':
    main()

Upvotes: 0

Related Questions