CODEWITHSUNDEEP
pythonmultiprocessing
JohnDoe
JohnDoe

Reputation: 643

Run large python script in parallel/multiprocessing

I have a large python script (an economic model with rows > 1500) which I want to excecute in parallel on several cpu cores. All the examples for multiprocessing I found so far were about simple functions, but not whole scripts. Could you please give me a hint how to achieve this? Thanks!

Clarification: the model generates as an output a dataset for a multitude of variables. Each result is randomly different from the other model runs. Therefore I have to run the model often enough till some deviation measure is achieved (let's say 50 times). Model input is allways the same, but not the output.

Edit, got it:

import os                                                                       
from multiprocessing import Pool                                                

n_cores = 4
n_iterations = 5                                 

def run_process(process):                                                             
    os.system('python myscript.py')                                       

if __name__ == '__main__':
    p = Pool(n_cores)
    p.map(run_process, range(n_iterations))

Upvotes: 0

Views: 843

Answers (1)

Patol75
Patol75

Reputation: 4547

If you want to use a pool of workers, I usually do the following.

import multiprocessing as mp


def MyFunctionInParallel(foo, bar, queue):
    res = foo + bar
    queue.put({res: res})
    return


if __name__ == '__main__':
    data = []
    info = {}
    num = 
    ManQueue = mp.Manager().Queue()
    with mp.Pool(processes=numProcs) as pool:
        pool.starmap(MyFunctionInParallel, [(data[v], info, ManQueue)
                                   for v in range(num)])
resultdict = {}
for i in range(num):
    resultdict.update(ManQueue.get())

To be clearer, your script becomes the body of MyFunctionInParallel. This means that you need to slightly change your script so that the variables which depend on your input (i.e. each of your models) can be passed as arguments to MyFunctionInParallel. Then, depending on what you want to do with the results you get for each run, you can either use a Queue as sketched above or for example, write your results in a file. If you use a Queue, it means that you want to be able to retrieve your data at the end of the parallel execution (i.e. in the same script execution), and I would advise to use dictionaries as a way to store your results in the Queue, as they are very flexible on the data they can contain. On the other hand, writing up your results in a file is I guess better if you wish to share them with other users/applications. You have to be careful with concurrent writing from all the workers, so as to produce a meaningful output, but writing one file per model can also be OK.

For the main part of the code, num would be the number of models you will be running, data and info some parameters which are specific (or not) to each model and numProcs the number of processes that you wish to launch. For the call to starmap, it will basically map the arguments in the list comprehension to each call of MyFunctionInParallel, allowing each execution to have different input arguments.

Upvotes: 1

Related Questions