CODEWITHSUNDEEP
windowspython-3.xmultiprocessing
Alpha the White
Alpha the White

Reputation: 13

Why aren't my Python Multiprocessing worker processes using multiple cores?

Using the Pool class from multiprocessing, I am splitting a database search task into parallel processes, each running a set of regular expressions against a very large database which I have loaded into memory. The program runs on a nice beefy Windows server with 60+ cores and plenty of memory.

My Python programming experience, and especially Multiprocessing, is pretty surface level.

When I first created the program, everything worked just fine, each worker processed its piece just fine and continued on to the next one. I didn't touch it for a couple months until I had to make some formatting changes to the database queries, but when I fired it up again it was running much too slowly. In testing, I determined that the number of processes I spawned didn't actually change the operating speed, and indeed looking at the task manager shows all the processes chilling in there, but only one of them is actually showing any signs of work.

def calc(ruleList,record):
    returnList = []
    print(record[5],end = '\r')
    hits = recordIterator(ruleList,record)
    for h in hits:
        returnList.append([record[0],record[1],h])
    return returnList

nthreads = 48
hname = 'Hits.txt'
p = multiprocessing.Pool(processes = nthreads)
Hits = []
for record in Records:
    Hits.append((p.apply_async(calc, (rules, record))).get())

hhandle = open(hname, "w")
for hit in Hits:
    try:
        for x in hit:
            hhandle.write(str(x[0])+'|'+str(x[1])+'|'+str(x[2])+'\n')
    except (UnicodeEncodeError,UnicodeDecodeError):
        pass
hhandle.close()

I am not an administrator on the machine, and I am unfamiliar with how to configure a server, but it appears to me that Windows is simply not scheduling the child processes to separate cores. I have tried reconfiguring my code in a number of different ways to avoid potential multiprocessing blockages, but each functional variation ends up suffering the same problem.

Is there something in my code that I've missed that's hobbling the processes? Is there some Windows Server setting that may have been changed to disqualify my workers from using separate cores?

Upvotes: 1

Views: 457

Answers (2)

johntellsall
johntellsall

Reputation: 15160

To flesh out @blckknght's answer: the apply_async() submits a job, but the .get() demands the results immediately. A simpler solution is to submit all the jobs, then take each result as they come in, regardless of order. That is, use imap_unordered()

source

import multiprocessing

def calc(num):
    return num*2

pool = multiprocessing.Pool(5)
for output in pool.imap_unordered(calc, [1,2,3]):
    print 'output:',output

output

output: 2
output: 4
output: 6

Upvotes: 0

Blckknght
Blckknght

Reputation: 104682

It looks to me like the code (p.apply_async(calc, (rules, record))).get() is forcing your program to run only one job at a time. The parent process will wait in get() for the result of the previous job to become available before launching the next job.

Try replacing the loop over Records and multiple apply_async calls with one call to starmap:

Hits = p.starmap(calc, ((rules, record) for record in Records))

This will pass of the records to the pool, and only after they've all been sent block for the results to come in.

Upvotes: 2

Related Questions