CODEWITHSUNDEEP
bashshelloptimizationparallel-processinggnu
wtz
wtz

Reputation: 373

Replacing nested while loop with GNU Parallel yields slow results

Simple nested loop:

while read f1; do
    while read f2; do
    ipv4comb+=($f1,f2)
    done < $src2
done < $src1

$src1 has 88 lines
$src2 has 175 lines
produced output has 15400 lines

$src1 is retrieved via ssh so lookup varies slightly, but, this loop usually yields:

./run.sh 0.34s user 0.18s system 15% cpu 3.327 total

If i replace the loop with:

parallel -j+0 echo {1} {2} :::: $temp_ipv4 $MM_BLACKLIST_IPV4 | wc -l

It yields:

./run.sh 49.31s user 31.27s system 242% cpu 33.239 total

What am i doing wrong?

Upvotes: 1

Views: 95

Answers (1)

Ole Tange
Ole Tange

Reputation: 33685

GNU Parallel has an overhead of ~2-10 ms per job, so 15400 jobs in 49s is within the expected range.

There are some tricks to deal with short running jobs. They, however, do not apply to your case: https://www.gnu.org/software/parallel/man.html#EXAMPLE:-Speeding-up-fast-jobs

Upvotes: 1

Related Questions