CODEWITHSUNDEEP
pythonparallel-processing
Rajesh Alayil
Rajesh Alayil

Reputation: 1

How to do parallel computing in python 2.7 using MPI?

I have written a python code to carry out genetic algorithm optimization, but it is too slow. I would like to know how to run the same in parallel mode making use of multiple CPUs ?

For more clarity, another python code will be called by my code for, say 100 times one after the other, I wanted to divide this between 4 CPUs. So that 25 times the outside python code is solved by each CPU. Thereby increasing the speed. Its highly appreciated if someone can help me with is ?

Thanks in advance!

Upvotes: 0

Views: 465

Answers (1)

Mike McKerns
Mike McKerns

Reputation: 35207

There are several packages that provide parallel computing for python2. I am the author of a package called pathos, which provides parallel computing with several parallel backends and gives them a common API. pathos provides parallel pipes and maps for multi-process, multi-threaded, parallel over sockets, MPI-parallel, and also interactions with schedulers and over ssh. pathos relies on several packages, which you can pick from if you don't want all the different options.

pathos uses: pyina which in turn uses mpi4py. mpi4py provides bindings to MPI, but you can't run the code from python 'normally'… you need to run with whatever you use to run MPI. pyina enables you to run mpi4py from normal python, and to interact with schedulers. Plus, pyina uses dill, which can serialize most python objects, and thus you are much more able to send what you want across processes.

pathos provides a fork of multiprocessing that also plays well with dill and pyina. Using both can enable you to do hierarchical parallel competing -- like launching MPI parallel that then spawns multiprocess or multithreaded parallel.

pathos also uses ppft, which is a fork of pp (Parallel Python), which provides parallel computing across sockets -- so that means you can connect a parallel map across several machines.

There are alternatives to pathos, such as IPython-parallel. However, the ability to use MPI is very new, and I don't know how capable it is yet. It may or may not leverage IPython-cluster-helper, which has been in development for a little while. Note that IPython doesn't use pp, it uses zmq instead, and IPython also provides connectivity to EC2 if you like cloud stuff.

Here are some relevant links:

  1. pathos, 'pyina, dill, ppft: https://github.com/uqfoundation
  2. Ipython-cluster-helper: https://github.com/roryk/ipython-cluster-helper
  3. IPython: http://ipython.org/ipython-doc/dev/parallel/parallel_mpi.html

Upvotes: 2

Related Questions