CODEWITHSUNDEEP
bashparallel-processinggnu-parallel
branquito
branquito

Reputation: 4044

Running bash script using gnu parallel

I have my script using while read to process some file line by line..

When I do:

head -n5 file1 | ./myscript.sh

I get my results well.

But trying to parallelize it using gnu parallel:

head -n5 file1 | parallel -j 4 ./myscript.sh

yields result file empty!?

I tried also with:

parallel -j 4 -a file1 ./myscript.sh

but still doesn't work. I was trying to do similar to what they say in documentation, but without any success. What am I doing wrong?

EDIT:

Maybe this can help:

head -n5 file1 | parallel -a - -j 4 echo #this works
head -n5 file1 | parallel -a - -j 4 ./myscript #this doesn't

Upvotes: 10

Views: 9082

Answers (2)

Ole Tange
Ole Tange

Reputation: 33685

--pipe is made for you:

cat file1 | parallel --pipe -N5 ./myscript.sh

But you need to change myscript.sh so it does not save to result but instead print the output to stdout. Then you can:

cat file1 | parallel --pipe -N5 ./myscript.sh > result

and avoid any mixing.

Upvotes: 5

gandaliter
gandaliter

Reputation: 10101

parallel doesn't send the lines of input to stdin of the command given to it, but appends the line to the command you give.

If you write it like you have, then you're effectively calling ./myscript.sh <INPUT>, where you want to call ./myscript.sh, and send the input as stdin.

This should work:

head -n5 file1 | parallel -j 4 "echo {} | ./myscript.sh"

The {} indicates to parallel where you want the input to go, rather than the default of at the end.

Upvotes: 8

Related Questions