CODEWITHSUNDEEP
matlabunixcluster-computingsimulationgnu-parallel
CMN
CMN

Reputation: 31

Run parallel simulations in remote unix cluster from matlab

I want to run simulations in parallel in a remote Cluster calling them from matlab.

I manage to run them in my local Ubuntu machine using.

unix('parallel -j4 flow > /dev/null :::: Pool.txt');

But when I want it to run it in a remote cluster I really did not mange to make the parallel command to work.

The first problem was to avoid entering the password.

For that I used the sshpass as this

unix('sshpass -p password ssh [email protected]')

That get me in to the server but it does not continue to the next command line.

I try so many commands that I do not want to reference here.

But basically can some one that understands well the parallel GNU command usage tell me how can I connect to a cluster. and run the simulations there. is it better just to make a script at the server and run it from matlab?

Any expert advice is highly appreciated.

Upvotes: 1

Views: 166

Answers (1)

Mark Setchell
Mark Setchell

Reputation: 207465

Your problem is not with GNU Parallel but with configuring ssh. First, you must get ssh set up, then the rest is easy.

So, on your local Ubuntu machine, you need to create your keys:

ssh-keygen -t rsa -b 2048

That will make some files in $HOME/.ssh. You now need to copy the public part of those keys to each and every node of the remote cluster where you want to run your parallel jobs:

ssh-copy-id -i $HOME/.ssh/id_rsa.pub CLUSTERUSERNAME@NODE-0
...
ssh-copy-id -i $HOME/.ssh/id_rsa.pub CLUSTERUSERNAME@NODE-15

e.g. 

ssh-copy-id -i $HOME/.ssh/id_rsa.pub [email protected]

Now, test you can ssh into each node:

ssh fred@node2

Then, on your local Ubuntu box, set up your config file for ssh, so it will be $HOME/.ssh/config

Host node0
Hostname 192.168.0.100
User fred

...
...

Host node15
Hostname 192.168.0.115
User fred

Now you can just use:

ssh node0

and it will know that means [email protected]

Now GNU Parallel will work with:

parallel -S node0,node1,node2

Upvotes: 1

Related Questions