CODEWITHSUNDEEP
bash
Kian
Kian

Reputation: 3229

How do I run a maximum of 4 jobs in the background?

I use bash scripting to make use of multi cores by running C++ programs in the background using the & symbol. (I'm a mathematician doing Monte Carlo Simulations).

This way I can run a C++ program in the background many times. For example:

for x in $(seq 1 1 100)
do 
./program_name &
done

Sometimes program_name will run fast, and sometimes it will run slow. This is random and impossible to know in advance. This prompts my question:

How do I make it, so that at any given time, no more than 4 jobs will be running simultaneously? Lets suppose 4 jobs are running at the moment. I want it so as soon as one finishes, the next one starts.

Upvotes: 2

Views: 85

Answers (3)

Petr Skocik
Petr Skocik

Reputation: 60067

( #Start a subshell
  N=4 #Specify number of jobs
  for((i=1;i<100;i++)) 
   do [ $((i%N)) = 0 ] && wait #Wait every N jobs
    (sleep 1; echo "$i";)&      #Some task
   done
 wait 
)

The logic is pretty simple and it can be turned into a one-liner. You need a subshel to prevent interference with processes running in the background of the current session that we don't want to wait on.

Upvotes: 0

Kian
Kian

Reputation: 3229

The other solutions to this post are better than the one I am showing here. But I'd like to document my "less than perfect one" for completition. It doesn't have the ability to add a new job to the queue after one finishes... which is a bad thing because it creates idle time.

But the advantage is that you don't need to install anything. I'm hoping also that this is going to be less bug prone.

count=1 # current number of jobs processing
j=4 ## maximum number of jobs

for x in $(seq 1 1 10)
do 
 echo Currently running programe $x
 ## CPP code goes here
 sleep $x &  ## Pretend it takes x seconds to complete

 count=$((count+1))  ## add one to the no. jobs running  

  if [ "$count" -gt "$j" ]; then
  echo "Waiting for jobs to finish"
  wait
  count=1
  fi
done

A clearly inefficient method, and I welcome any improvements.

Upvotes: 1

that other guy
that other guy

Reputation: 123490

On GNU, FreeBSD, and OSX you can use xargs:

echo {1..100} | xargs -n 1 -P 4 ./program_name

Alternatively, you can install and use GNU parallel, which is made for exactly this kind of thing (for 4 jobs rather than 1 job/core, add -j 4):

parallel --gnu ./program_name ::: {1..100}

Upvotes: 1

Related Questions