CODEWITHSUNDEEP
pythonparallel-processingmultiprocessing
mxdbld
mxdbld

Reputation: 17755

Blocking multiprocessing pool.map called in a process

I want to call a multiprocessing.pool.map inside a process.

When initialized inside the run() function, it works. When initialized at instantiation, it does not.

I cannot figure the reason for this behavior ? What happens in the process ? I am on python 3.6

from multiprocessing import Pool, Process, Queue

def DummyPrinter(key):
    print(key)

class Consumer(Process):
    def __init__(self, task_queue):
        Process.__init__(self)
        self.task_queue = task_queue
        self.p = Pool(1)

def run(self):
    p = Pool(8)
    while True:
        next_task = self.task_queue.get()
        if next_task is None:
            break

        p.map(DummyPrinter, next_task) #  Works
        #self.p.map(DummyPrinter, next_task)   #  Does not Work
    return

if __name__ == '__main__':
    task_queue = Queue()
    Consumer(task_queue).start()

    task_queue.put(range(5))
    task_queue.put(None)

Upvotes: 2

Views: 712

Answers (1)

noxdafox
noxdafox

Reputation: 15060

multiprocessing.Pool cannot be shared by multiple processes because it relies on pipes and threads for its functioning.

The __init__ method gets executed in the parent process whereas the run logic belongs to the child process.

I usually recommend against sub-classing the Process object as it's quite counter intuitive.

A logic like the following would better show the actual division of responsibilities.

def function(task_queue):
    """This runs in the child process."""
    p = Pool(8)
    while True:
        next_task = self.task_queue.get()
        if next_task is None:
            break

        p.map(DummyPrinter, next_task) #  Works

def main():
    """This runs in the parent process."""
    task_queue = Queue()
    process = Process(target=function, args=[task_queue])
    process.start()

Upvotes: 2

Related Questions