CODEWITHSUNDEEP
bashshellsh
heresthebuzz
heresthebuzz

Reputation: 718

Display progress with script running in paralel

I have a bash script that runs in parallel inside a for loop, like this:

for (( n=0; n<${LIMIT}; n++ )); do
   my_function $n &

my_function is a function that runs in LIMIT virtual machines/nodes with ssh. It works fine, but i am not able to keep track of execution progress. It would be nice to have a progress bar or any other tool that can give a completion estimate. Unfortunately, all approaches i've searched doesn't apply to this parallel case.

Thanks

Upvotes: 0

Views: 487

Answers (2)

Charles Duffy
Charles Duffy

Reputation: 295470

The below script uses the widely-packaged 3rd-party tool pv ("pipe viewer") to write a progress bar.

pv reads input from stdin and writes it to stdout -- but can write a progress bar (and optionally, an ETA and other statistics) to stderr in the process. It's most commonly used in situations like curl | pv | tar, but is perfectly suitable for this use case as well.

#!/usr/bin/env bash

# version check: make sure automatic FD allocation is available
case $BASH_VERSION in
  ''|[1-3].*|4.[012].*) echo "ERROR: bash 4.3+ required" >&2; exit 1;;
esac

my_function() { sleep $(( 2 + (RANDOM % 10) )); } # replace with real function

# note "limit" is lowercase -- that's per POSIX guidelines
# see https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html
limit=100 # as an example
pids=( )

# allocate a file descriptor that writes to a copy of "pv" -- "pipe viewer"
exec {pv_fd}> >(exec pv --progress --size "$limit" >/dev/null); pv_pid=$!

for (( n=0; n<limit; n++ )); do
  # every time my_function completes, write to the copy of pv
  (my_function "$n" {pv_fd}>&-; rc=$?; printf '\n' >&$pv_fd; exit "$rc") & pids[n]=$!
done

exec {pv_fd}>&- # now, close the process substitution feeding to pv

# pause here until all copies of my_function have completed
# by looping and passing each copy of wait only one PID, we can retrieve exact
# exit status for each copy.
for n in "${!pids[@]}"; do
  pid=${pids[$n]}
  wait "$pid" || echo "WARNING: Process $n (PID $pid) exited with status $?" >&2
done
wait "$pv_pid" # and wait for pv to exit

echo "Finished running all instances of my_function" >&2

Upvotes: 4

Ole Tange
Ole Tange

Reputation: 33685

With GNU Parallel you can do this:

export -f my_function
seq 0 ${LIMIT} |
  parallel -j${LIMIT} --bar my_function

Upvotes: 0

Related Questions