CODEWITHSUNDEEP
bashloops
Liso
Liso

Reputation: 158

While loop fail to read all line from file

So I got this bash script that perform a backup command from remote server, each server hostname is defined on file called /tmp/quarantine.list. The content is like this:

server1
server2

server1 and server2 is an ssh alias to my remote server, so if I need to access either of them I just run e.g. ssh server1.

I need to backup each server in /tmp/quarantine.list, so I loop each line using while loop. Here is my code.

    # Reading from /tmp/quarantine.list
while IFS= read -r list; do
  if [ "$list" == "server4" ]; then
    users="adminjoe"
  fi
  tempfile="${logdir}/rsync_$list.log"
  SECONDS=0
  set +x
  if rsync -ca --stats --info=progress2 --remove-source-files $list:/home/${users}/backup/20* $backup_location/$list >> $tempfile 2>&1; then
    ssh $list "find /home/${users}/backup/20* -type d -empty -delete"
  else
    echo "Error while running rsync"
    exit 1
  fi
  set -x
  elapsed_time=$SECONDS
  transferred_size=$(gawk 'BEGIN { FS=":" } {gsub(/^[ \t]+|[ \t]+$/, "", $2)} /Total transferred file size/ { print $2 }' $tempfile | tr -cd "[:digit:]")
  echo "Backup of $list done at $(/bin/date +'%Y-%m-%d_%H:%M:%S') --> $(bytesToHumanReadable $transferred_size)" | tee -a "${HOME}/backup.log"
  echo "<b>$list --> $(bytesToHumanReadable $transferred_size)</b>" >> $konten
  echo "$transferred_size $list" >> $summaryfile_size
  echo "$elapsed_time $list" >> $summaryfile_time
done < "/tmp/quarantine.list"
rm "/tmp/quarantine.list"

The problem is my script only pick the first line of /tmp/quarantine.list, so for example if there was two remote in quarantine.

server2
server6

The script only process server2, while server6 remains untouched. What's weird is, when I put --dry-run option to rsync, it will process all line in /tmp/quarantine.list without problem !

Without --dry-run

+ echo 'Backup of server2 done at 2021-10-28_16:55:26 --> 3.28 GB'
Backup of server2 done at 2021-10-28_16:55:26 --> 3.28 GB
++ bytesToHumanReadable 3280769140
++ S=("bytes" "kB" "MB" "GB" "TB" "PB" "EB" "YB" "ZB")
++ local i=3280769140 d= s=0 S
++ (( i > 1000 && s < 9-1 ))
++ printf -v d .%02d 14
++ i=3280769
++ s=1
++ (( i > 1000 && s < 9-1 ))
++ printf -v d .%02d 76
++ i=3280
++ s=2
++ (( i > 1000 && s < 9-1 ))
++ printf -v d .%02d 28
++ i=3
++ s=3
++ (( i > 1000 && s < 9-1 ))
++ echo '3.28 GB'
+ echo '<b>server2 --> 3.28 GB</b>'
+ echo '3280769140 server2'
+ echo '1883 server2'
+ IFS=
+ read -r list
+ rm /tmp/quarantine.list

With --dry-run

+ echo 'Backup of server2 done at 2021-10-28_16:07:14 --> 3.28 GB'
Backup of server2 done at 2021-10-28_16:07:14 --> 3.28 GB
++ bytesToHumanReadable 3280769140
++ S=("bytes" "kB" "MB" "GB" "TB" "PB" "EB" "YB" "ZB")
++ local i=3280769140 d= s=0 S
++ (( i > 1000 && s < 9-1 ))
++ printf -v d .%02d 14
++ i=3280769
++ s=1
++ (( i > 1000 && s < 9-1 ))
++ printf -v d .%02d 76
++ i=3280
++ s=2
++ (( i > 1000 && s < 9-1 ))
++ printf -v d .%02d 28
++ i=3
++ s=3
++ (( i > 1000 && s < 9-1 ))
++ echo '3.28 GB'
+ echo '<b>server2 --> 3.28 GB</b>'
+ echo '3280769140 server2'
+ echo '32 server2'
+ IFS=
+ read -r list
+ '[' server6 == server4 ']'

As you can see, the latter continues with server6 as opposed of the former. I just wondering what I'd do wrong here?

Upvotes: 0

Views: 538

Answers (1)

glenn jackman
glenn jackman

Reputation: 246744

This can happen if the last line of the file does not end with a newline. Then read will actually read the line, but it will return a non-zero status (ref: https://www.gnu.org/software/bash/manual/bash.html#index-read). This non-zero status causes the while loop to end.

The idiomatic way to handle this is:

while IFS= read -r list || [[ -n $list ]]; do

Upvotes: 5

Related Questions