CODEWITHSUNDEEP
pythongoogle-compute-enginedistributed-computinggoogle-colaboratorympi4py
Darren Ng
Darren Ng

Reputation: 19

How can I run mpi4py code on Google Colab?

I have a piece of code written with tasks distributed using the mpi4py library. The code in essence is just distributing loops and running them in multiple core which cannot be vectorise, and it looks something like this:

from mpi4py import MPI
comm = MPI.COMM_WORLD
numtasks = comm.Get_size()
taskid = comm.Get_rank()
MASTER = 0
TAG1 = 1
TAG2 = 2
TAG3 = 3
TAG4 = 4

Usually when I have to run on my own local machine, I'd just type into my terminal something like

mpiexec -np 4 python my-program-name.py

if I was to run on all 4 cores. My question is how would I go about doing the same thing but on Google VMs, like Google Colab? I'm looking for a layman instruction for doing distributed programming using mpi4py on google VMs basically.

The only thing I can find on stackoverflow is this Distributed Programming on Google Cloud Engine using Python (mpi4py) but it's now really helping. How do I do all these things such as "setting up cluster nodes via Google container engine" and "ssh-connect into one VM from my cluster and run the code"?

Upvotes: 0

Views: 5669

Answers (2)

madhava mk
madhava mk

Reputation: 11

Run commmand with sudo For example,sudo mpirun --allow-run-as-root --oversubscribe -np <num> <executable>

Upvotes: 0

sleepyhead
sleepyhead

Reputation: 469

In any Jupyter code window you can just type "!" followed by command, like so:

! pip install mpi4py

and it will be interpreted be ran like in terminal. So for you simply add "!" before the command you need to run: ! mpiexec -np 4 python my-program-name.py

This way you can do whatever you like: copy/create files, cloning repos...

Another option would be to use python subprocess library. Run this command as a sub process like so: import subprocess subprocess.call(['mpiexec', '-np', '4', 'python', 'my-program-name.py'])

Upvotes: 1

Related Questions