Locking mutex by multiple subprocesses

Question

Is the mutex doing it's job in this code? i.e is the sys.stdout.write protected?

import sys
from multiprocessing import Pool, Lock

class ParentApp():
    mutex=Lock()
    def report(self,msg):
        with ParentApp.mutex:
            sys.stdout.write(msg)

class ChildApp1(ParentApp):
    def print_report(self):
        for i in xrange(100):
            ParentApp.report(self, 'BLABLA')


class ChildApp2(ParentApp):
    def print_report(self):
        for i in xrange(100):
            ParentApp.report(self, 'TESTTEST')

def runnable(app):
    app.print_report()

def main():
    app=[]
    app.append(ChildApp1())
    app.append(ChildApp2())

    pool = Pool(len(apps))
    pool.map(runnable, apps)

    exit(0)

if __name__ == '__main__':
    sys.exit(main())

PS: code is also here : http://pastebin.com/GyV3w45F

PS: I am running on a Linux host

Answer 1

It is only doing it's job if you're on a Posix-like platform. If you're on Windows, each process ends up with a completely different copy of the lock.

You can see this by adding a little extra tracing and a sleep statement:

class ParentApp():
    mutex=Lock()
    def report(self,msg):
        print("\nGETTING for {}".format(msg))
        with self.mutex:
            print("GOT for {}".format(msg))
            sys.stdout.write(msg)
            sys.stdout.flush()
            time.sleep(5)

On Linux:

GETTING for BLABLA
GOT for BLABLA
BLABLA
GETTING for TESTTEST
< 5 second delay here>

On Windows:

GETTING for BLABLA
GOT for BLABLA
BLABLA
GETTING for TESTTEST
GOT for TESTTEST
TESTTEST
<5 second delay here>

This is because Posix platforms create new processes using os.fork() which means the Lock() you created in the parent process gets automatically inherited by the children. However, Windows doesn't have os.fork, so it needs to spawn a new process, then re-import your module in the child. Re-importing the module means ParentApp gets re-imported and re-executed, and so does Lock class attribute. So, your parent and two children each end up with their own unique Lock.

To fix this, you need to create a single Lock in the parent, and pass that to the children. This is actually not a trivial task with your current architecture - you're passing the Lock object as an argument to pool.map, which won't allow you to pass Lock objects. You'll get an exception if you try it:

RuntimeError: Lock objects should only be shared between processes through inheritance

You can only pass normal Lock objects to children at the point you're actually starting a Process. Once they're started (like they are when you call a Pool method), you get that exception:

l = Lock()
p = Process(target=func, args=(l,)) # ok
p.start()

pool = Pool()
pool.apply(func, args=(l,)) # not ok, raises an exception.

In order to pass a Lock to a Pool function like map, you need to use a multiprocessing.Manager to create a shared lock. Here's what I would recommend doing:

import sys 
from multiprocessing import Pool, Lock, get_context, Manager
import time

class ParentApp():
    def __init__(self, mutex):
        self.mutex = mutex

    def report(self,msg):
        with self.mutex:
            sys.stdout.write(msg)

class ChildApp1(ParentApp):
    def print_report(self):
        for i in range(100):
            ParentApp.report(self, 'BLABLA')


class ChildApp2(ParentApp):
    def print_report(self):
        for i in range(100):
            ParentApp.report(self, 'TESTTEST')

def runnable(app):
    app.print_report()

def main():
    apps=[]
    m = Manager()
    lock = m.Lock()
    apps.append(ChildApp1(lock))
    apps.append(ChildApp2(lock))

    pool = Pool(len(apps))
    pool.map(runnable, apps)

if __name__ == '__main__':
    sys.exit(main())

In order to make sure the Lock is shared, we need to have ParentApp actually take the lock object as an argument. This isn't a nice as having it entirely self-contained in the class, but I think that's the best we can do with the limitations of Windows.