CODEWITHSUNDEEP
bashgnugnu-parallel
alpha_cod
alpha_cod

Reputation: 2033

Run script in multiple machines in parallel

I am interested to know the best way to start a script in the background in multiple machines as fast as possible. Currently, I'm doing this

Run for each IP address

ssh user@ip -t "perl ~/setup.pl >& ~/log &" &

But this takes time as it individually tries to SSH into each one by one to start the setup.pl in the background in that machine. This takes time as I've got a large number of machines to start this script on.

I tried using GNU parallel, but couldn't get it to work properly:

seq COUNT | parallel -j 1 -u -S ip1,ip2,... perl ~/setup.pl >& ~/log

But it doesn't seem to work, I see the script started by GNU parallel in the target machine, but it's stagnant. I don't see anything in the log.

What am I doing wrong in using the GNU parallel?

Upvotes: 4

Views: 5883

Answers (3)

Eran Ben-Natan
Eran Ben-Natan

Reputation: 2615

Try to wrap ssh user@ip -t "perl ~/setup.pl >& ~/log &" & in the shell script, and run for each ip address ./mysctipt.sh &

Upvotes: 0

Ole Tange
Ole Tange

Reputation: 33685

GNU Parallel assumes per default that it does not matter which machine it runs a job on - which is normally true for computations. In your case it matters greatly: You want one job on each of the machine. Also GNU Parallel will give a number as argument to setup.pl, and you clearly do not want that.

Luckily GNU Parallel does support what you want using --nonall:

http://www.gnu.org/software/parallel/man.html#example__running_the_same_command_on_remote_computers

I encourage you to read and understand the rest of the examples, too.

Upvotes: 3

itzhaki
itzhaki

Reputation: 840

I recommend that you use pdsh

It allows you to run the same command on multiple machines

Usage: 

pdsh -w machine1,machine2,...,machineN <command>

It might not be included in your distribution of linux so get it through yum or apt

Upvotes: 2

Related Questions