CODEWITHSUNDEEP
pythonpandasdatabasemultiprocessingdata-storage
Krishna Mojamdar
Krishna Mojamdar

Reputation: 1

How to store data during multipreocessing in python

I have a process in which an optimiser which runs thousands of iterations with different inputs.

optimizer = ng.optimizers. NGOpt (parametrization=instrum, budget=10000, num workers = 25)

with futures. ProcessPoolExecutor (max workers=optimizer.num_workers) as executor:

   recommendation optimizer.minimize (run_model, verbosity = 0, executor = executor, batch_mode=False)

Here I'm trying to minimise the return value of the function run model which is having the dataframes I want to log. The optimiser is using 25 workers as shown in the code

Currently multiprocessing is integrated with this process to reduce run time and also scale up the number of iterations.

The problem I have here is that, during each of the process I need to log the dataframes in a consolidated form so that I can monitor the performance of the optimiser in real time.

I keep running into process failures when if I use any I/o files

Created a sqllite3 database and append values to the tables in them as an interim solution .Although I'm aware this is not a full proof answer but is the only solution that I could think off.

Not able to incorporate multithreading as well due to compatibility issues and encountering GIL

Looking for any suggestions or methods to store these multiple dataframes during multipreocessing

Upvotes: 0

Views: 48

Answers (0)

Related Questions