CODEWITHSUNDEEP
bashunixssh
Pat Mustard
Pat Mustard

Reputation: 1902

Running sequential commands through ssh on multiple servers

I have a list of servers in a file that I pass to the following script:

#!/bin/bash

while IFS='' read -r line || [[ -n "$line" ]]; do

    ssh_cmd="user@$line"
    ssh $ssh_cmd 'hostname && ls -l /path/to/folder | grep some_token' &

done < "$1"

What I want is output like this:

  output from hostname (1)
  output from ls -l (1)
  output from hostname (2)
  output from ls -l (2)
  output from hostname (3)
  output from ls -l (3)

But these seem to be mixed up:

  output from ls -l (1)
  output from hostname (1)      
  output from hostname (2)
  output from ls -l (2)
  output from ls -l (3)
  output from hostname (3)

Is there some way to ensure that these commands happen sequentially?

Thanks in adv!

Upvotes: 0

Views: 1325

Answers (2)

Mark Setchell
Mark Setchell

Reputation: 208003

Use GNU Parallel to do them all nicely in parallel and use the -k option to keep the output in order:

parallel -k -a hosts.txt 'echo ssh user@{} "hostname; ls"'

So, if hosts.txt contains the following:

host1
host27
host32
host100
hostfreddyfrog
hostmichaelfish

you will get this:

ssh user@host1 hostname; ls
ssh user@host27 hostname; ls
ssh user@host32 hostname; ls
ssh user@host100 hostname; ls
ssh user@hostfreddyfrog hostname; ls
ssh user@hostmichaelfish hostname; ls

Obvioulsy you need to remove the echo and add in your quoting and gripping, but the concept should be clear enough.

Another advantage, other than it now being a one-liner, is that if you have 32,000 hosts to work on, GNU Parallel can be told, using the -j flag, how many processes to run at a time without overloading your server. By default, if you have 8 cores in your CPU, it will run 8 jobs in parallel, but you could set it to 64 for example with

parallel -k -j 64 ...

You can also set up GNU Parallel to be able to ssh into hosts itself, normally in order to distribute processing jobs amongst servers, then the job becomes even easier.

parallel -S node1,node4,node56 'hostname; grep ...'

See example here.

ALternative syntaxes for passing hosts.txt are

parallel -k 'echo ssh user@{} "hostname; ls"' < hosts.txt

or

parallel -k 'echo ssh user@{} "hostname; ls"' :::: hosts.txt

Upvotes: 0

Maximillian Laumeister
Maximillian Laumeister

Reputation: 20399

The & at the end of this line makes it so the process runs in the background:

ssh $ssh_cmd 'hostname && ls -l /path/to/folder | grep some_token' &

If you remove the & from that line, the script will wait until the ssh session finishes to continue on to the next iteration of the loop. Also, you will need to add the -n flag to the ssh command so that it doesn't read from stdin. This will give you the sequential output you're looking for.

Final command:

ssh -n $ssh_cmd 'hostname && ls -l /path/to/folder | grep some_token'

Upvotes: 1

Related Questions