CODEWITHSUNDEEP
pythonpython-3.xsubprocessmultiprocessing
tjm
tjm

Reputation: 7550

Appending a process to a list (but doing nothing with it) alters program behaviour

In the following program, when I append the process to a list (a seemingly pointless thing to do), it runs as expected. But if I remove the append, the processes destructor is called many times before it is even run. There are only n constructions but (n)(n+1)/2 (where n is the number of processes) destructions. This leads me to believe that each process is being copied into each new process and then immediately destroyed. Perhaps that is how the multiprocessing module works. It makes sense, since each process is a fork of the current one. But, what is the significance of appending to the list? Why does simply doing that stop this behaviour?

Here's the test and sample output:

import multiprocessing

class _ProcSTOP:
    pass

class Proc(multiprocessing.Process):

    def __init__(self, q, s): 
        s.i += 1
        self._i = s.i 
        self._q = q 
        print('constructing:', s.i)
        super().__init__()

    def run(self):
        dat = self._q.get()

        while not dat is _ProcSTOP:
            self._q.task_done()
            dat = self._q.get()

        self._q.task_done()
        print('returning:   ', self._i)

    def __del__(self):
        print('destroying:  ', self._i)



if __name__ == '__main__':

    q = multiprocessing.JoinableQueue()
    s = multiprocessing.Manager().Namespace()
    s.i = 0 
    pool = []

    for i in range(4):
        p = Proc(q, s)
        p.start()
        pool.append(p)    # This is the line to comment

    for i in range(10000):
        q.put(i)

    q.join()

    for i in range(4):
        q.put(_ProcSTOP)

    q.join()

    print('== complete')

Sample output with append:

constructing: 1
constructing: 2
constructing: 3
constructing: 4
returning:    3
returning:    2
== complete
returning:    1
returning:    4
destroying:   4
destroying:   3
destroying:   2
destroying:   1

Sample output without append:

constructing: 1
constructing: 2
constructing: 3
destroying:   1
constructing: 4
destroying:   1
destroying:   2
destroying:   3
destroying:   1
destroying:   2
returning:    1
returning:    4
returning:    2
== complete
returning:    3
destroying:   1
destroying:   3
destroying:   2
destroying:   4

Upvotes: 8

Views: 273

Answers (1)

Mihai Stan
Mihai Stan

Reputation: 1052

Adding the object to a list will prevent it from being deleted in child processes because after forking it will call "os._exit()" instead of clearing the whole stack

The relevant code is in multiprocessing/forking.py (constructor of Popen, which is called on "p.start()")

 self.pid = os.fork()
 if self.pid == 0:
     if 'random' in sys.modules:
         import random
         random.seed()
     code = process_obj._bootstrap()
     sys.stdout.flush()
     sys.stderr.flush()
     os._exit(code)

Where _bootstrap setups the new process and calls "run" (code is in multiprocessing/process.py)

Upvotes: 1

Related Questions