CODEWITHSUNDEEP
bash
sherif
sherif

Reputation: 2332

Parallel execution for programs for each file in dir in bash

How can I divide work on all files in a directory to several cpus using bash without using the parallel command

Upvotes: 2

Views: 1562

Answers (3)

damienfrancois
damienfrancois

Reputation: 59110

The xargs command together with the find command is often used in this situation; like this:

find . -maxdepth 1 -type f -print0 | xargs -0 -n 1 -P 4 echo

You'll need to replace the 4 with the maximum number of simultaneous processes you want (typically the same as the number of physical cores in your computer for CPU-bound tasks) and echo with the actual name of the program you want to run on those files.

The option maxdepth prevents find from recursing in the directory (remove it if you want recursion), and the pair -print0/-0 is there to handle word splitting safely in the output of find (typically to guard against filenames with spaces in them)

Upvotes: 2

sherif
sherif

Reputation: 2332

NR_CPUS=3
format=.jpg
job_count=0
for f in *."$format"; do
    mycommand "$f" &
    job_count=$((job_count+1))
    if [ "$job_count" -eq $NR_CPUS ]; then
        wait
        job_count=0
    fi
done

Upvotes: 3

Sebastian
Sebastian

Reputation: 1428

You can spawn a seperate process for each file (that's what parallel does... much more cleverly)

 for f in *.jpg; do
   some_program_or_shell_function $f &
 done

Upvotes: 1

Related Questions