CODEWITHSUNDEEP
bashparallel-processingbatch-processingbackground-process
Uko
Uko

Reputation: 13396

How limiting background jobs works?

while looking on how to parallelize bash tasks, I've stumbles over a code like this:

for item in "${items[@]}"
do
  ((i=i%THREADS)); ((i++==0)) && wait
  process_item $item &
done

Where process_item is some king of function/program that works with item and the THREADS var contain the maximum number of background processes that can run simultaneously.

Can someone explain to me how this works? I understand that i=i%THREADS ensures that i is between 0 and THREADS-1, and that i++==0 increments i and checks whether it is 0. But is wait bound to all sub processes? Or how does it know that is has to wait until the previous batch stopped processing?

Upvotes: 4

Views: 1259

Answers (2)

chepner
chepner

Reputation: 532093

It's an obfuscated way of writing

for item in "${items[@]}"
do
    # Every THREADSth job, stop and wait for everything
    # to complete.
    if (( i % THREADS == 0 )); then
        wait
    fi
    ((i++))
    process_item $item &
done

It also doesn't actually work terribly well. It doesn't ensure that there are always $THREADS jobs running, only that no more than $THREADS jobs are running at once.

Upvotes: 1

Renaud Pacalet
Renaud Pacalet

Reputation: 29290

i++==0 checks and increments, not the opposite. wait waits for all currently active child processes. So, each iteration (but the first, thanks to the ((i++==0))) first waits for the process launched by the previous iteration and launches a new process.

Upvotes: 0

Related Questions