CODEWITHSUNDEEP
pythonmultiprocessingipythonpyzmq
Phil G
Phil G

Reputation: 171

Python multiprocessing hanging on pool.join()

I'm having problems with Python freezing when I try to use the multiprocessing module. I'm using Spyder 2.3.2 with Python 3.4.3 (I have previously encountered problems that were specific to iPython).

I've reduced it to the following MWE: import multiprocessing

def test_function(arg1=1,arg2=2):
    print("arg1 = {0}, arg2 = {1}".format(arg1,arg2))
    return None

pool = multiprocessing.Pool(processes=3)
for i in range(6):
    pool.apply_async(test_function)

pool.close()
pool.join()

This, in its current form, should just produce six identical iterations of test_function. However, while I can enter the commands with no hassle, when I give the command pool.join(), iPython hangs, and I have to restart the kernel.

The function works perfectly well when done in serial (the next step in my MWE would be to use pool.apply_async(test_function,kwds=entry).

for i in range(6):
    test_function()
arg_list = [{'arg1':3,'arg2':4},{'arg1':5,'arg2':6},{'arg1':7,'arg2':8}]
for entry in arg_list:
    test_function(**entry)

I have (occasionally, and I'm unable to reliably reproduce it) come across an error message of ZMQError: Address already in use, which led me to this bug report, but preceding my code with either multiprocessing.set_start_method('spawn') or multiprocessing.set_start_method('forkserver') doesn't seem to work.

Can anyone offer any help/advice? Thanks in advance if so.

Upvotes: 4

Views: 3847

Answers (2)

carthurs
carthurs

Reputation: 553

@Anarkopsykotik is correct: you must use a main, and you can get it to print by returning a result to the main thread.

Here's a working example.

import multiprocessing
import os

def test_function(arg1=1,arg2=2):
    string="arg1 = {0}, arg2 = {1}".format(arg1,arg2) +" from process id: "+ str(os.getpid())
    return string

if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=3)
    for i in range(6):
        result = pool.apply_async(test_function)
        print(result.get(timeout=1))
    pool.close()
    pool.join()

Upvotes: 2

Anarkopsykotik
Anarkopsykotik

Reputation: 517

Two things pop to my mind that might cause problems. First, in the doc, there is a warning about using the interactive interpreter with multiprocessing module : https://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers

Functionality within this package requires that the main module be importable by the children. This is covered in Programming guidelines however it is worth pointing out here. This means that some examples, such as the Pool examples will not work in the interactive interpreter.

Second: you might want to retrieve a string with your async function, and then display it from your main thread. I am not quite sure child threads have access to standard output, which might be locked to the main thread.

Upvotes: 1

Related Questions