CODEWITHSUNDEEP
pythonoopparallel-processingsimpy
mivandev
mivandev

Reputation: 321

Exporting data from an objected oriented SimPy model that is run with parallel processing

I have created a simulation model in SimPy using OO principles. The main logic is contained in class Model. Among others, it contains an Entity Generator function that generates entities to flow through the model. During runtime, output is saved using Model.save_data() in list output_list, which is created once outside of any class.

To reduce computation times when I run multiple runs, I want to benefit from using multiple CPU cores. To execute my model without using parallel processing, I use the following code:

for run in range(g.number_of_runs):
    print(f"Run {run + 1}")
    my_model = Model(run)
    my_model.run()

I have tried to use the joblib library to execute my for-loop in a parallel way. I first wrote a wrapper function:

import time
from joblib import Parallel, delayed

def wrapper(run):
    print(f"Run {run + 1}")
    my_model = Model(run)
    my_model.run()

results = Parallel(n_jobs=16)(delayed(wrapper)(run) for run in range(g.number_of_runs))

This code executes without error. The model seems to run as I input a print() statement to check. However, output_list remains empty as nothing is stored in it. Executing multiple runs without parallel processing and storing the data of each run in output_list works as expected. What am I missing here?

Upvotes: 0

Views: 105

Answers (1)

mivandev
mivandev

Reputation: 321

The following SO answer helped me figured it out: How to append items to a list in a parallel process (python)?

With the following code, I am able to execute my SimPy model in parallel and save the data for each run to a list:

if (__name__ == '__main__'):
    output_list = mp.Manager().list()
    results = Parallel(n_jobs=16)(delayed(wrapper)(run) for run in range(g.number_of_runs))

    # Convert the Manager().list() list to a regular list
    output_list_pure = list(output_list)

Upvotes: 1

Related Questions