CODEWITHSUNDEEP
pythonpython-multiprocessing
Mike
Mike

Reputation: 2747

Python multiprocessing subclass initialization

Is it okay to initialize the state of a multiprocessing.Process subclass in the __init__() method? Or will this result in duplicate resource utilization when the process forks? Take this example:

from multiprocessing import Process, Pipe
import time

class MyProcess(Process):
    def __init__(self, conn, bar):
        super().__init__()
        self.conn = conn
        self.bar = bar
        self.databuffer = []

    def foo(self, baz):
        return self.bar * baz

    def run(self):
        '''Process mainloop'''
        running = True
        i = 0
        while running:
            self.databuffer.append(self.foo(i))
            if self.conn.poll():
                m = self.conn.recv()
                if m=='get':
                    self.conn.send((i, self.databuffer))
                elif m=='stop':
                    running = False
            i += 1
            time.sleep(0.1)


if __name__=='__main__':
    conn, child_conn = Pipe()
    p = MyProcess(child_conn, 5)
    p.start()      
    time.sleep(2)

    # Touching the instance does not affect the process which has forked.
    p.bar=1
    print(p.databuffer)

    time.sleep(2)
    conn.send('get')
    i,data = conn.recv()
    print(i,data)
    conn.send('stop')
    p.join()

As I note in the code, you cannot communicate with the process via the instance p, only via the Pipe so if I do a bunch of setup in the __init__ method such as create file handles, how is this duplicated when the process forks?

Does this mean that subclassing multiprocessing.Process in the same way you would a threading.Thread a bad idea?

Note that my processes are long running and meant to handle blocking IO.

Upvotes: 2

Views: 1311

Answers (1)

Wayne Werner
Wayne Werner

Reputation: 51817

This is easy to test. In __init__, add the following:

 self.file = open('does_it_open.txt'.format(self.count), 'w')

Then run:

 $ strace -f python youprogram.py 2> test.log
 $ grep does_it_open test.log
 open("does_it_open.txt", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 6

That means that at least on my system, copying your code and adding that call, the file was opened once, and only once.

For more about the wizardry that is strace, check out this fantastic blog post.

Upvotes: 2

Related Questions