How to run multiple dd commands in background & see status?

Question

I want to run multiple dd commands in background, but be able to see the status.

I have the following script.sh:

#!/usr/bin/env bash

for drive in $@
do
  echo "Wiping $drive"
  dd if=/dev/zero of=$drive status=progress &
done
wait
echo "Done."

Which results in the following output:

$ sudo bash ./script.sh /dev/sda /dev/sdb
Wiping /dev/sda
Wiping /dev/sdb
288788992 bytes (289 MB, 275 MiB) copied, 10 s, 28.9 MB/s 14404864 bytes (114 MB, 109 MiB) copied, 4 s, 28.6 MB/s

Is there any way how to output the respective dd statuses below the drive paths? For example:

$ sudo bash ./script.sh /dev/sda /dev/sdb
Wiping /dev/sda
288788992 bytes (289 MB, 275 MiB) copied, 10 s, 28.9 MB/s
Wiping /dev/sdb
14404864 bytes (114 MB, 109 MiB) copied, 4 s, 28.6 MB/s

I tried various redirects, named pipes etc. but wasn't able to achieve such (or similar) output.

---

I tried the coprocesses approach which seems to be the way to go, but now I'm unable to make it work with the for cycle.

This works fine:

coproc dd_sda { dd if=/dev/zero of=/dev/sda status=progress 2>&1; }
echo "sda PID: $dd_sda_PID"
coproc dd_sdb { dd if=/dev/zero of=/dev/sdb status=progress 2>&1; }
echo "sdb PID: $dd_sdb_PID"

sda PID: 12494
./wipe.sh: line 86: warning: execute_coproc: coproc [12494:dd_sda] still exists
sdb PID: 12496

However this:

for drive in sda sdb
do
  coproc_name=dd_${drive}
  coproc $coproc_name { dd if=/dev/zero of=/dev/$drive status=progress 2>&1; }
  pid_var="${coproc_name}_PID"
  echo "$drive PID: ${!pid_var}"
done

doesn't work for the second coprocess:

sda PID: 12759
./wipe.sh: line 39: warning: execute_coproc: coproc [12759:dd_sda] still exists
sdb PID: 

When hardcoding the name using if condition, it also works:

for drive in sda sdb
do
  coproc_name=dd_${drive}
  if [[ "$drive" == 'sda' ]]
  then
    coproc dd_sda { dd if=/dev/zero of=/dev/$drive status=progress 2>&1; }
  elif [[ "$drive" == 'sdb' ]]
  then
    coproc dd_sdb { dd if=/dev/zero of=/dev/$drive status=progress 2>&1; }
  fi
  pid_var="${coproc_name}_PID"
  echo "$drive PID: ${!pid_var}"
done

sda PID: 12998
./wipe.sh: line 39: warning: execute_coproc: coproc [12998:dd_sda] still exists
sdb PID: 13000

Answer 1

Here is my final solution, massively inspired by @Diego Torres Milano (thanks again).

#!/usr/bin/env bash

drives=$@
number_of_drives=$#
drives_to_wipe=($drives)

stripped_drive_path() {
  echo $1 | awk -F/ '{ print $3 }'
}

wipe() {
  echo "Filling \`$1\` drive with zeros:"
  dd if=/dev/zero of=$1 status=progress 2>&1
}

echo "Total number of drives: $number_of_drives"

# Wipe the drives in parallel

iteration=1
for drive in $drives; do
  drive_name=$(stripped_drive_path $drive)
  coproc_name=dd_${drive_name}
  eval coproc $coproc_name "{ wipe $drive; }"
  if [[ "$iteration" == 1 ]]; then
    echo 'Feel free to ignore the following warnings'
  fi
  ((iteration++))
done

# Display the progress

iteration=1
# Run until all drives are wiped
while [[ ${#drives_to_wipe[@]} > 0 ]]; do
  for drive in $drives; do
    drive_name=$(stripped_drive_path $drive)
    coproc_name=dd_${drive_name}

    # Move one line below the "Filling drive with zeros" message
    if [[ $iteration > 1 ]]; then
      tput cud 1
    fi

    # Read the drive's current status from the coprocess
    if read -r -d $'\r' -u ${!coproc_name[0]} line &> /dev/null; then
      tput el # Clear line
      echo -e "$line\r\n"
    else
      # Remove the finished drive from the list
      for i in ${!drives_to_wipe[@]}; do
        if [ "${drives_to_wipe[$i]}" == "$drive" ]; then
          unset drives_to_wipe[$i]
        fi
      done
      tput el # Clear line
      echo -e "$(($number_of_drives - ${#drives_to_wipe[@]}))/$number_of_drives done!\r\n"
    fi

    # Move back one line up
    if [[ $iteration > 1 ]]; then
      tput cuu 1
    fi

  done

  # Move two lines up for each drive
  if [[ ${#drives_to_wipe[@]} > 0 ]]; then
    tput cuu $(($number_of_drives * 2))
  fi

  ((iteration++))
done

echo "All drives ($number_of_drives) wiped."

Which results in such output:

$ sudo bash ./script.sh /dev/sda /dev/sdb
Total number of drives: 2
Feel free to ignore the following warnings
./wipe.sh: line 24: warning: execute_coproc: coproc [85994:dd_sda] still exists
Filling `/dev/sda` drive with zeros:
1/2 done!
Filling `/dev/sdb` drive with zeros:
2/2 done!
All drives (2) wiped.

Answer 2

Using coprocesses you can do something like this

#! /bin/bash

coproc DD1 { dd if=/dev/zero of=$drive1 status=progress 2>&1; }
coproc DD2 { dd if=/dev/zero of=$drive2 status=progress 2>&1; }
coproc DD3 { dd if=/dev/zero of=$drive3 status=progress 2>&1; }

el=$(tput el)
cuu1=$(tput cuu1)

IFS=
while :
do
    for n in {1..3}
    do
        v="DD$n"
        if read -r -d $'\r' -u ${!v[0]} line
        then
            printf '%s%s: %s\r\n' "$el" "$v" "$line"
        else
            printf '%s: Done\r\n' "$el" "$v"
        fi
    done
    for n in {1..3}
    do
        printf '%s' "$cuu1"
    done
done

my dd does not have status so I'm assuming dd adds a \r instead of \n.

edit

for loop (unfortunately eval seems to be the only option, don't know why coproc is not take the name from a variable though)

for n in {1..3}
do
    eval coproc "DD${n}" "{ dd if=/dev/zero of=\$drive$n status=progress 2>&1; }"
done

Answer 3

Redirect the status of each dd to a file, and print those files repeatedly while clearing the old output.

dd status=progress from GNU coreutils prints its status to stderr, so use 2> to redirect the status info.
The status is updated by overwriting the current line with \r followed by the new status. Since \r only works for single lines, updating four different lines needs terminal control sequences, e.g. ANSI escape codes which can be printed conveniently using commands like clear and tput.

outfile_prefix=/tmp/wipe-status-$$-
for drive in "$@"; do
  (
    echo "Wiping $drive"
    # dummy-version for testing ...
    dd if=/dev/zero of=/dev/null bs=1 count=5M status=progress 2>&1
    # ... if you are happy with the output, replace it with
    # dd if=/dev/zero of="$drive" status=progress 2>&1
  ) > "$outfile_prefix$BASHPID" &
done

clear -x  # clear the currently visible screen
while             # do-while loop
  tput home                  # move cursor to top left
  sed '' "$outfile_prefix"*  # print files like `cat`, but with \n at the end
  jobs %% &> /dev/null       # while jobs are running
do
  sleep 1
done

rm "$outfile_prefix"*

Above code is inefficient if you are wiping many big or slow drives, because the status files keep growing (even though they seem to contain only one line) and we keep printing those growing files over and over again.

If you run into problems, try to increase the sleep time. Maybe sed 's/.*\r//' instead of sed '' can speed things up. tail -c100 would definitively help, but inserts the temp. file names into the output.

The correct way to handle situation would be ...

• Use fifos instead of files (see mkfifo command).
• Remove the status=progress and periodically ...
  • Call kill -s USR1 for all* running dd jobs,
    which causes them to print a single status line.
  • Update some internal status array by reading the fifos*.
  • Print all statuses from the array.

* At some point, only a few dd jobs will run while others already finished and closed their fifos, which complicates that process a bit. This is the main reason I sticked with files for this answer.