CODEWITHSUNDEEP
pythonlinuxmultiprocessing
user1265125
user1265125

Reputation: 2656

Force-utilizing multiple cores

So I have an executable which I need to run a number of times with different input parameters. It takes around an hour per run, and I've noticed during the entire hour CPU utilisation of only 1 core out of 8 is at 100%, rest of them are idling.

Is it possible to spawn 4-5 processes, each using a different core and working on different parameters? Is this common practice? This is the first time I'm bothering about multiple cores so if there are any other things I need to be aware of please let me know.

Currently I'm using Python to run and wait for this executable to finish.

Upvotes: 2

Views: 83

Answers (3)

Matteo Italia
Matteo Italia

Reputation: 126967

What you describe is routinely done when compiling big programs - multiple compiler processes are spawned, working on different files.

If your program is CPU bound as it seems, the input data is easily partitionable and the various instances wouldn't stomp on each other's feet when writing the results, you can try and see if you obtain the expected speedup.

Upvotes: 2

Karol Nowak
Karol Nowak

Reputation: 662

One way to approach this, as has already been suggested, was to partition your input set in the application and process it in parallel using the multiprocessing module.

Alternatively you can partition the input up-front and run multiple copies of your program on the inputs using GNU parallel or good old xargs (look at -n and -P options).

Then there is the problem of merging the results back together, if needed.

All of this hinges on being able to split inputs into parts that can be processed independently, without coordination or shared memory. Otherwise it gets more complex.

Upvotes: 1

Vasfed
Vasfed

Reputation: 18504

When running parallel processes shared resources should be taken into consideration, plus depending on load profile it may or may not be faster than single process(for example if bottleneck is not the cpu)

Common problems are usually related to "race conditions" and deadlocks, former is the case when two processes work with same data not knowing about each other so that data gets corrupted due to overwrites for example

Not knowing more details about the task it's impossible to answer exactly

Upvotes: 1

Related Questions