CODEWITHSUNDEEP
bash
urzeit
urzeit

Reputation: 2909

Parallel executing commands and wait

I have a set of scripts, e.g.

01_some_stuff1
02_some_stuff2
03_some_stuff3a
03_some_stuff3b
04_some_stuff4a
04_some_stuff4b

These scripts should run ordered by their number and scripts with the same number should run in parallel.

My first idea was to iterate the possible numbers

for n in $(seq -f %02.0f 0 99); do
    for s in "${n}_*"; do
        export CURRENT_JOB="${s}"
        "${s}" &
    done
    wait
done

Is this a safe method? Is there a more elegant solution that also allows to set a different environment for the inner loop elements?

Upvotes: 0

Views: 77

Answers (2)

chepner
chepner

Reputation: 530853

The only real change you need is to avoid quoting the * in your pattern. If you are using bash 4.0 or later, you can use brace expansion to eliminate the dependency on seq.

# for n in $(seq -f %02.0f 0 99); do
for n in {00..99}; do
    for s in "${n}"_*; do
        export CURRENT_JOB="${s}"
        "${s}" &
    done
    wait
done

Upvotes: 2

Mark Setchell
Mark Setchell

Reputation: 207345

You could use GNU Parallel like this:

#!/bin/bash

# Don't barf if no matching files when globbing
shopt -s nullglob

for n in $(printf "%02d " {1..4}); do
   # Get list (array) of matching scripts
   scripts=( ${n}_* )
   if [ ${#scripts[@]} -gt 0 ]; then
      parallel --dry-run -k 'CURRENT_JOB={} ./{}' ::: ${scripts[@]}
   fi
   echo barrier
done

Sample Output

CURRENT_JOB=01_some_stuff1 ./01_some_stuff1
barrier
CURRENT_JOB=02_some_stuff2 ./02_some_stuff2
barrier
CURRENT_JOB=03_some_stuff3a ./03_some_stuff3a
CURRENT_JOB=03_some_stuff3b ./03_some_stuff3b
CURRENT_JOB=03_some_stuff3c ./03_some_stuff3c
barrier
CURRENT_JOB=04_some_stuff4a ./04_some_stuff4a
CURRENT_JOB=04_some_stuff4b ./04_some_stuff4b
barrier

Remove the echo barrier and --dry-run to actually run it properly.

Upvotes: 3

Related Questions