CODEWITHSUNDEEP
python-3.xpython-multiprocessing
magladde
magladde

Reputation: 634

How to run bash scripts in parallel

I have a series of 250 bash scripts which need to be executed in parallel. Each script utilizes approximately 1 core so I do not want to execute them all at the same time. I would like to run 10 scripts at the same time and whenever one finishes execute another script.

Upvotes: 0

Views: 61

Answers (1)

Paul Hodges
Paul Hodges

Reputation: 15293

I recommend parallel, but I'm going to post this monstrosity for the benefit of having people pick it apart and tune it. :)

#! /bin/env bash

## TODO: this documentation block needs to be expanded.
use="

  $0 <#procs> <cmdfile>

  Pass the number of desired processes to prespawn as the 1st argument.
  Pass the command file with the list of tasks you need done.

  Command file format:
   KEYSTRING:cmdlist

  where KEYSTRING will be used as a unique logfile name
  and   cmdlist   is the base command string to be run

  KEYSTRING may not contain whitespace of any sort.
  Other lines are not allowed, including blanks or comments.

"

die()  { echo "$@ $use" >&2; exit 1; }

case $# in
2) case "$1" in
   *[^0-9]*) die "INVALID #procs '$1'" ;;
   esac
   declare -i primer="$1"  # a countdown of how many processes to pre-spawn
   cmdfile="$2"
   [[ -r "$cmdfile" ]]  || { die "$cmdfile not readable"; }
   grep -v : "$cmdfile" || { die "$cmdfile has invalid lines"; }
   declare -i lines=$( grep -c : $cmdfile)
   if (( lines < primer ))
   then die "Note - command lines in $cmdfile ($lines) fewer than requested process chains ($primer)"
   fi  ;;
*) die ;;
esac >&2

trap 'echo abort $0@$LINENO; use; exit 1' ERR       # make sure any error is fatal
trap ': no-op to ignore' HUP  # ignore hangups (built-in nohup without explicit i/o redirection)

spawn() {
  IFS="$IFS:" read key cmd && [[ "${cmd:-}" ]] || return
  echo "$(date) executing '$cmd'; c.f. $key.log" | tee "$key.log"
  echo "# autogenerated by $0 $(date)
   { $cmd
     spawn
   } >> $key.log 2>&1 &
  " >| $key.sh
  . $key.sh
  rm -f $key.sh
  return 0
}

# create a command list based on those designators
declare chains=0
while (( primer-- ))  # until we've filled the requested quota
do spawn              # create a child process
done < $cmdfile

Upvotes: 1

Related Questions