CODEWITHSUNDEEP
pythonmultithreadingconcurrencydependencies
Miquel
Miquel

Reputation: 858

Multiprocessing with python and dependencies

I have a program with 10 functions, and I want to take profit of concurrency to make it more efficient. I have extracted the dependencies between functions, which are the following:

f1 <- f2,f3,f4,f5
f2 <- f6
f3 <- f7,f8,f9
f4 <- f10
f5 <- f10
f8 <- f10
f9 <- f10

Can I achieve this with the multiprocessing library?

Can anybody give me an snippet of code to start from it?

My question is quite similar to this one, but I would like to get it using the build in Python libraries.

Parallel Tasking Concurrency with Dependencies on Python like GNU Make

Thanks,

Upvotes: 1

Views: 2589

Answers (1)

Kyle Dewey
Kyle Dewey

Reputation: 690

I'm not 100% sure what you're asking for, but any task of this sort can be approached in the following way. Given the dependencies you provided, a dependency graph can be constructed using topological sort. Using this information, it is possible to immediately determine which nodes have no dependencies, i.e. those without incoming edges. These nodes can all be processed in parallel. Once a node has been processed, all descendent nodes can be marked that the given dependency has been satisfied. Once all dependencies for a node have been satisfied, then the node can be run.

In your case, running a node means performing a function call. As such, instead of simply marking that the dependency has been satisfied, you may want to store the result of the function call.

As an aside, without seeing the functions, this may or may not actually yield any performance benefit. Generally, this sort of parallelism is too fine-grained; more time is spent performing parallel coordination as opposed to actually running work in parallel.

---EDIT---

I wrote a small Scala library that I believe performs what you want. Unfortunately, a similar elegant solution isn't possible in CPython, since it doesn't support proper multithreading. It's still possible but clumsy; the whole framework would need to be written in a master-slave fashion. This also limits parallelism, as the master acts as a bottleneck for the processing of new tasks.

Upvotes: 1

Related Questions