CODEWITHSUNDEEP
bashfor-loopparallel-processing
k k
k k

Reputation: 33

Is there a way to parallelize a bash for loop?

I have a simple script that pulls the SMART data from a series of hard drives and writes it to a timestamped log file which is later logged and parsed for relevant data.

filename="filename$( date '+%Y_%m_%d_%H%M' ).txt"
for i in {a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p}
do
smartctl -a /dev/sd$i >> /path/to/location/$filename
done

Since this takes several seconds to run, I would like to find a way to parallelize it. I've tried just appending an '&' to the end of the single line in the loop, however that causes the text file to be written haphazardly as sections finish rather than sequentially and in a readable manner. Is there a way to fork this into seperate processes for each drive then pipe the output back into an orderly text file?

Also, I assume setting the filename variable will have to be moved into the for loop in order for the forks to be able to access it. That causes an issue however if the script runs long enough to roll over into a new minute (or two) and then the script becomes sequentially datestamped fragments rather than one contiguous file.

Upvotes: 3

Views: 1328

Answers (2)

Mark Setchell
Mark Setchell

Reputation: 207465

With GNU Parallel like this:

parallel -k 'smartctl -a /dev/{}' ::: a b c d e f g h  i j k l m n o p > path/to/output

The -k option keeps the output in order. Add -j 8 if you want to run, say, 8 at a time, else it will one per core at a time. Or -j 16 if you want to run them all at once...

parallel -j 16 -k 'smartctl ....

Of course, if you are in bash you can do this too:

parallel -j 16 -k 'smartctl -a /dev/{}' ::: {a..o} > path/to/output

Upvotes: 2

Gilles
Gilles

Reputation: 9489

Wouldn't something like this work? (not tested)

filename="filename$( date '+%Y_%m_%d_%H%M' ).txt"
for i in {a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p}
do
smartctl -a /dev/sd$i > /path/to/location/$filename.$i &
done
wait
cat /path/to/location/$filename.* > /path/to/location/$filename

EDIT: it looks like the final cat is slow, so what about this version?

filename="filename$( date '+%Y_%m_%d_%H%M' ).txt"
tmpdir="/dev/shm/tmp$( date '+%Y_%m_%d_%H%M' )"
mkdir $tmpdir
for i in {a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p}
do
smartctl -a /dev/sd$i > $tmpdir/$filename.$i &
done
wait
cat $tmpdir/$filename.* > /path/to/location/$filename
rm -rf $tmpdir

Upvotes: 1

Related Questions