CODEWITHSUNDEEP
pythonmultiprocessing
Amaranth
Amaranth

Reputation: 2511

multiprocessing function call

I am trying to do some multiprocessing in Python. I have a function doing some work and returning a list. I want to repeat that for several cases. Finally, I want to get the returned list of each parallel call and unify them (have only one list with all duplicates removed).

def get_version_list(env):
    list = []
    #do some intensive work
    return list

from multiprocessing import Pool

pool = Pool()

result1 = pool.apply_async(get_version_list, ['prod'])
result2 = pool.apply_async(get_version_list, ['uat'])
#etc, I have six environment to check.

alist = result1.get()
blist = result2.get()

This does not work (not sure about the function call syntax, but I tried other things too, without success), it gives me an error (and repeats it a lot since my intensive work is doing around 300 request.post calls).

RuntimeError: Attempt to start a new process before the current process has finished its bootstrapping phase.

This probably means that you are on Windows and you have
forgotten to use the proper idiom in the main module:

    if __name__ == '__main__':
        freeze_support()
        ...

The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce a Windows executable.

Upvotes: 0

Views: 637

Answers (1)

scomes
scomes

Reputation: 1846

You have to put the multiprocessing portion inside of your main function, like:

def get_version_list(env):
    list = []
    print "ENV: " + env
    return list


if __name__ == '__main__':
    from multiprocessing import Pool

    pool = Pool()

    result1 = pool.apply_async(get_version_list, ['prod'])
    result2 = pool.apply_async(get_version_list, ['uat'])
    #etc, I have six environment to check.

    alist = result1.get()
    blist = result2.get()

Upvotes: 1

Related Questions