CODEWITHSUNDEEP
gnu-parallel
Chris
Chris

Reputation: 31206

GNU Parallel: How to recieve stdin from gnu parallel pipe as if it were from a file?

Suppose I have the following command which runs like so: 

my_command -f file

How do I set up parallel such that content piped in is executed as if it were a pipe?

# For example, in pseudo code: 

cat item | parallel my_command -f <(the incoming stdin)

It is important to do it this way, as I am processing specific chunks of data via the --speadstdin option, so if there as a way to redirect the piped input as if it were coming from a file, that would solve for the situation.

Upvotes: 1

Views: 446

Answers (1)

Ole Tange
Ole Tange

Reputation: 33685

You are looking for --cat or --fifo:

cat file | parallel --pipe --cat my_command -f {}

Faster, but requires my_command read the complete file from start to end once and only once:

cat file | parallel --pipe --fifo my_command -f {}

Faster, but requires file is a real file:

parallel -a file --pipepart --block -1 --fifo my_command -f {}

See also chapter 9.6 in GNU Parallel 2018 (Online: https://doi.org/10.5281/zenodo.1146014 Printed: http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html).

Upvotes: 3

Related Questions