CODEWITHSUNDEEP
bashshell
IsaIkari
IsaIkari

Reputation: 1154

How to wait until all background jobs initiated in a while loop have all finished a certain steps?

I'd like to start a sequence of background job in a while loop in a bash script:

END=10
i=0;
while [ $i -lt $END ]
do
(SOME COMMAND;\
sleep 3;
SOME COMMAND 1;\
SOME COMMAND 2;\
SOME COMMAND 3;) &
i=`expr $i + 1`
done

And outside this while loop I'd like to wait until all these 10 background job all finished SOME COMMAND 1 and then proceed, so the psudo code is something like:

until [the number of process that finished SOME COMMAND 1 is greater or equal to $END];
do
sleep 0.5
done

What I tried to do is to add another counter like this:

true_started_cnt=0
END=10
i=0;
while [ $i -lt $END ]
do
(SOME COMMAND;\
sleep 3;
SOME COMMAND 1;\
true_started_cnt=`expr $true_started_cnt + 1`;\
SOME COMMAND 2;\
SOME COMMAND 3;) &
i=`expr $i + 1`
done

until [ $true_started_cnt -ge $END ]
do
echo "Waiting for initial period for all streams to finish"
sleep 0.5
done

But this seems not working, possibly because of simultaneous write of the same global var by multiple bg jobs. I wonder how to achieve my intention in this piece of code.

Referring to this post, I'm also trying:

echo 1 >/dev/shm/foo

END=10
i=0;
while [ $i -lt $END ]
do
(SOME COMMAND;\
sleep 3;
SOME COMMAND 1;\
echo $(($(</dev/shm/foo)+1)) >/dev/shm/foo;\
SOME COMMAND 2;\
SOME COMMAND 3;) &
i=`expr $i + 1`
done

until [ $(echo $(</dev/shm/foo)) -ge $END ]
do
  echo "Waiting, true_started_cnt=$(echo $(</dev/shm/foo))"
  sleep 1
done

But still not working.

Upvotes: 2

Views: 1505

Answers (3)

Ionuț G. Stan
Ionuț G. Stan

Reputation: 179119

You could use a FIFO queue to signal completion of a certain stage of jobs:

rendezvous.sh
#!/usr/bin/env bash

shopt -so errexit
shopt -so nounset

declare -ri job_count=5

# A FIFO queue where jobs will signal their completion.
mkfifo rendezvous

# Clean up FIFO queue on script exit.
trap EXIT EXIT; EXIT() {
  rm -f rendezvous
  trap  - EXIT      # Restore default interrupt handler.
  kill -s EXIT "$$" # Propagate interrupt.
}

# Wait until the known number of jobs have signaled their completion to the
# FIFO queue.
wait_rendezvous() {
  (
    local -i counter=0

    while read -r; do
      (( ++counter ))

      if (( counter == job_count )); then
        break
      fi
    done
  ) < rendezvous
}

# Signal completion to the FIFO queue.
show_at_rendezvous() {
  # For some reason, without the trailing ampersand some A jobs end up
  # dangling.
  echo "done" > rendezvous &
}

#
# Actual script logic starts below.
#
command_a() {
  local -i job_id=$1
  echo "Command A[$job_id]: start"
  sleep 5
  show_at_rendezvous
  echo "Command A[$job_id]: done"
}

command_b() {
  local -i job_id=$1
  echo "Command B[$job_id]: start"
  sleep 10
  echo "Command B[$job_id]: done"
}

# Launch jobs.
for i in $(seq 1 "$job_count"); do
  command_a "$i" &
  command_b "$i" &
done

# Wait for all A jobs to signal their completion.
wait_rendezvous
echo "All A commands done; now awaiting for B commands..."

# Wait for all remaining jobs to finish.
wait
echo "All B commands done."

Output

$ ./rendezvous.sh 
Command A[1]: start
Command B[1]: start
Command A[2]: start
Command B[2]: start
Command A[3]: start
Command B[3]: start
Command A[4]: start
Command B[4]: start
Command A[5]: start
Command B[5]: start
Command A[2]: done
Command A[1]: done
All A commands done; now awaiting for B commands...
Command A[5]: done
Command A[3]: done
Command A[4]: done
Command B[1]: done
Command B[2]: done
Command B[3]: done
Command B[4]: done
Command B[5]: done
All B commands done.

Upvotes: 0

konsolebox
konsolebox

Reputation: 75478

Recording progress isn't needed if all you want is to make sure all processes exit.

#!/bin/bash

END=10
pids=()

for (( i = 0; i < END; ++i )); do
    ( sleep "$(( RANDOM % 10 + 1 ))" ) &
    pids[$!]=$!
done


while [[ ${#pids[@]} -gt 0 ]]; do
    for pid in "${pids[@]}"; do
        echo "Waiting for $pid to exit."
        wait "$pid"
        kill -s 0 "$pid" >/dev/null || unset 'pids[pid]'
    done

    sleep 1
done

Upvotes: 0

Philippe
Philippe

Reputation: 26452

There could be race condition in your last script as well, try this :

cp /dev/null /dev/shm/foo

END=10
for ((i=0; i<$END; i++)); do
    (
    echo SOME COMMAND;\
    sleep $((i+1));\
    echo $i SOME COMMAND 1;\
    echo $i >>/dev/shm/foo;\
    echo $i SOME COMMAND 2;\
    sleep $((i+1));\
    echo $i SOME COMMAND 3
    ) &
done

until [ $(wc -l /dev/shm/foo | awk '{print $1}') -ge $END ]
do
    echo "Waiting, /dev/shm/foo contains $(echo $(</dev/shm/foo))"
    sleep 1
done

echo "*********** All SOME COMMAND 1 finished"
wait
echo "Script finished"

Upvotes: 1

Related Questions