CODEWITHSUNDEEP
linuxmultithreadingbashshell
alturic
alturic

Reputation: 53

Bash, run multiple commands simultaneously, wait for N to finish before spawning more

Alright, so I've tried gnu parallel and there's some quirks about getting that to work that makes it not possible for me to use.

Ultimately I'd love to just be able to do something like this:

for modelplot in /${rfolder}/${mplotsfolder}/${mname}/$mscript.gs
do

for regionplot in $mregions
do
opengrads -bclx "${modelplot}.gs $regionplot ${plotimage} ${date} ${run} ${gribfile} ${modelplot}" && wait -n
done

done

But I can't seem to find a way to limit the spawning of background processes to a specific number. Someone mentioned doing:

for i in {1..10}; do echo "${i} & (( count ++ > 5 )) && wait -n; done

Should do it, but I can't really verify if it is working that way. It seems like it just spawns them all instantly. I'm assuming the output in terminal of that should be: echo 1, echo 2, echo 3, echo 4, echo 5. Then echo 6, echo 7, echo 8, echo 9, echo 10.

I'm just trying to spawn, say 5, iterations of a loop, wait for those to finish and then spawn the next 5, wait for those to finish, spawn the next 5, etc until the loop is done.

Upvotes: 2

Views: 708

Answers (2)

Ole Tange
Ole Tange

Reputation: 33748

From your comments it is pretty unclear what you want.

But using the {= =} construct you can get almost anything in the arguments.

Append .gs only on the first run:

parallel echo {}'{= if($job->seq() == 1) { $_ = ".gs" } else { $_="" } =}' ::: a b c

Append .gs only on the last run:

parallel echo {}'{= if($job->seq() == $Global::JobQueue->total_jobs() ) { $_ = ".gs" } else { $_="" } =}' ::: a b c

Disregarding the comments and looking only at the loop in the question the solution is:

parallel --header : opengrads -bclx "{modelplot}.gs {regionplot} ${plotimage} ${date} ${run} ${gribfile} {modelplot}" ::: modelplot /${rfolder}/${mplotsfolder}/${mname}/$mscript.gs ::: regionplot $mregions

Upvotes: 1

chepner
chepner

Reputation: 532333

Each time you start a background job, increment a count. When that count reaches 5 (or whatever), wait for all background jobs to complete, then reset the count to 0 and resume starting background jobs.

p_count=0
for modelplot in /${rfolder}/${mplotsfolder}/${mname}/$mscript.gs; do
  for regionplot in $mregions; do
      opengrads -bclx "${modelplot}.gs $regionplot ${plotimage} ${date} ${run} ${gribfile} ${modelplot}" &
      if (( ++p_count == 5 )); then
          wait
          p_count=0
      fi
    done
  done
done

It is surprisingly tricky to keep exactly 5, rather than at most 5, jobs running in the background in shell. (wait -n lets you know when a job has finished, but not how many have finished.) To keep the machine busy, a tool like xargs or parallel is more appropriate.

Upvotes: 4

Related Questions