CODEWITHSUNDEEP
bashscp
Matias Barrios
Matias Barrios

Reputation: 5056

SCP loop stops executing after some time

So I have these two versions of the same script. Both are attempting to copy my profile to all the servers on my infra ( about 5k ). The problem I am having is that no matter which version I use, I always get the process stuck somewhere around 300 servers. It does not matter if I do it sequentially or in parallel, both version fail and both at a random server. I dont get any error message (Yes I know Im redirecting error messages to null now), it simply stops executing after reaching a random point close to 300 servers and it just lingers there doing nothing.

The best run I could get did it for about 357 servers.

Probably there is some detail I unknow that is causing this. Could someone advise?

Sequential

#!/bin/bash
clear
echo "$(date) - Process started"
all_count="$( cat all_servers.txt | wc -l )"
while read server
do
    scp -B -o "StrictHostKeyChecking no" ./.bash_profile rouser@${server}:/home/rosuer/ && echo "$server - Done!" >> ./log.log || echo "$server - Failed!" >> ./log.log 

done <<< "$( cat all_servers.txt )"


echo "$(date) - Process completed!!"

Parallel

#!/bin/bash
clear
echo "$(date) - Process started"
all_count="$( cat all_servers.txt | wc -l )"
while read server
do
    scp -B -o "StrictHostKeyChecking no" ./.bash_profile rouser@${server}:/home/rosuer/ && echo "$server - Done!" >> ./log.log || echo "$server - Failed!" >> ./log.log &

done <<< "$( cat all_servers.txt )"

wait

echo "$(date) - Process completed!!"

Upvotes: 0

Views: 1242

Answers (1)

Adam Katz
Adam Katz

Reputation: 16186

Let's start with better input parsing. Instead of parsing a bash herestring from a posix command substitution via a while read loop, I've got the while read loop running through your server list directly via pipeline (this assumes one server per line in that file. I can fix this if that's not the case). If the contents of all_servers.txt was too long for a command line, you'd experience an error and/or premature termination.

I've also removed extraneous ./ items and I assume that rouser's home directory on each server is in fact /home/rouser (scp defaults to the home directory if given a relative path or no path at all).

Sequential

#!/bin/bash
clear
echo "$(date) - Process started"
all_count="$( cat all_servers.txt | wc -l )"

while read server
do
    scp -B -o "StrictHostKeyChecking no" .bash_profile rouser@${server}: \
        && echo "$server - Done!" >> log.log \
        || echo "$server - Failed!" >> log.log 
done < all_servers.txt

echo "$(date) - Process completed!!"

Parallel

For the Parallel solution, I've enclosed your conditional in parentheses just in case the pipeline was backgrounding the wrong process.

#!/bin/bash
clear
echo "$(date) - Process started"
all_count="$( cat all_servers.txt | wc -l )"

while read server
do
    (
        scp -B -o "StrictHostKeyChecking no" .bash_profile rouser@${server}: \
          && echo "$server - Done!" >> log.log
          || echo "$server - Failed!" >> log.log
    ) &
done < all_servers.txt

wait

echo "$(date) - Process completed!!"

SSH keys

I highly recommend learning more about SSH. The scp -B flag was unknown to me because I'm used to using SSH keys and ssh-agent, which will make such connectivity seamless (use passwordless keys if you're running this in a cron job).

Upvotes: 2

Related Questions