CODEWITHSUNDEEP
pythonmultiprocessing
markk
markk

Reputation: 737

Python multiprocessing prevent switching off to other processes

While using the multiprocessing module in Python, is there a way to prevent the process of switching off to another process for a certain time?

I have ~50 different child processes spawned to retrieve data from a database (each process = each table in DB) and after querying and filtering the data, I try to write the output to an excel file.

Since all the processes have similar steps, they all end up to the writing process at similar times, and of course since I am writing to a single file, I have a lock that prevents multiple processes to write on the file.

The problem is though, the writing process seems to take very long, compared to the writing process when I had the same amount of data written in a single process (slower by at least x10)

I am guessing one of the reasons could be that while writing, the cpu is constantly switching off to other processes, which are all stuck at the mutex lock, only to come back to the process that is the only one that is active. I am guessing the context switching is a significant waste of time since there are a lot of processes to switch back and forth from

I was wondering if there was a way to lock a process such that for a certain part of the code, no context switching between processes happen

Or any other suggestions to speed up this process?

Upvotes: 0

Views: 1247

Answers (2)

shevron
shevron

Reputation: 3673

Don't use locking and don't write from multiple processes; Let the child processes return the output to the parent (e.g. via standard output), and have it wait for the processes to join to read it. I'm not 100% on the multiprocessing API but you could just have the parent process sleep and wait for a SIGCHLD and only then read data from an exited child's standard output, and write it to your output file.

This way only one process is writing to the file and you don't need any busy looping or whatever. It will be much simpler and much more efficient.

Upvotes: 1

Zimm3r
Zimm3r

Reputation: 3425

You can raise the priority of your process (Go to Task Manager, right click on the process and raise it's process priority). However the OS will context switch no matter what, your process has no better claim then other processes to the OS.

Upvotes: 0

Related Questions