CODEWITHSUNDEEP
pythonpython-2.7numpyparallel-processingmultiprocessing
abhinavkulkarni
abhinavkulkarni

Reputation: 2409

Incorrect output when using multiprocessing

I am doing gradient descent (100 iterations to be precise). Each data point can be analyzed in parallel, there are 50 data points. Since I have 4 cores, I create a pool of 4 workers using multiprocessing.Pool. The core of the program looks like following:

# Read the sgf files (total 50)
(intermediateBoards, finalizedBoards) = read_sgf_files()

# Create a pool of processes to analyze game boards in parallel with as 
# many processes as number of cores
pool =  Pool(processes=cpu_count())

# Initialize the parameter object
param = Param()

# maxItr = 100 iterations of gradient descent
for itr in range(maxItr):
    args = []
    # Prepare argument vector for each file
    for i in range(len(intermediateBoards)):
        args.append((intermediateBoards[i], finalizedBoards[i], param))
    # 4 processes analyze 50 data points in parallel in each iteration of 
    # gradient descent
    result = pool.map_async(train_go_crf_mcmc, args)

Now, I haven't included definition for the function train_go_crf, but the very first line in the function is a print statement. So, when I execute this function the print statement should get executed 100*50 times. But that does not happen. What's more, I get different number of console outputs different number of times.

What's wrong?

Upvotes: 0

Views: 489

Answers (1)

tacaswell
tacaswell

Reputation: 87556

Your problem is that you are using map_async instead of map. This means that once all of the work is farmed out to the pool, it will continue on with the loop, even if all of the work has not been finished. It is not clear to me what will happen to the work still running when the next loop starts, but if these are supposed to be iterations, I can't imagine it is a) good b) well defined.

If you use map, it will block the loop until all of the worker functions have finished before moving on to the next step. I guess you could do this with sleep, but that would just make things more complicated for no gain. map will wait for exactly the minimum amount of time it needs to to let everything finish.

Upvotes: 1

Related Questions