CODEWITHSUNDEEP
bashparallel-processing
Matthew Bullock
Matthew Bullock

Reputation: 91

How to parallelise a while loop in bash?

I have the following while loop that runs in parallel. ( The logProcess is a function that I define earlier in my script. )

while read LINE; do
    logProcess $LINE &
done <<< "$ELS_LOGS"
wait

I need to find a way to limit the number of processes running. I know there are parallel processes running. How do I convert the loop to use that command?

Upvotes: 8

Views: 2834

Answers (2)

Charles Duffy
Charles Duffy

Reputation: 295281

GNU xargs is also an adequate tool for the job:

xargs -P 20 -d $'\n' -n 1 logProcess

...will run up to 20 concurrent logProcess instances, with each line of the stdin given to xargs passed to a different such instance.

Upvotes: 1

Mark Setchell
Mark Setchell

Reputation: 207355

May I recommend GNU Parallel to you. Your entire script would become:

parallel -a "$ELS_LOGS" logProcess

If logProcess is a function defined within your script, you will need to export it first before running GNU Parallel, like this:

export -f logProcess

Then if you want, say, 8 running at a time, you simply do:

parallel -j 8 -a "$ELS_LOGS" logProcess

If you want to see what it would do, without actually doing anything:

parallel --dry-run ...

If you want a progress bar, or an ETA:

parallel --eta ...
parallel --bar ...

Upvotes: 6

Related Questions