CODEWITHSUNDEEP
bashshellgnugnu-parallel
Andre Augusto
Andre Augusto

Reputation: 84

GNU Parallel: --line-buffer option doesn't work with --pipe

Im trying to parallelize some tasks that needs to be processed on real time, so i was using --line-buffer. I was processing very long strings, but then i noticed that sometimes it hits the line lenght limit, making a command line too long error, so i decided to pipe them

But when i use the --pipe option, --line-buffer stops working
I tested with simpler commands, and the issue still occurs

# Returns instantly, but pass the data as args
(echo 1; echo 2; sleep 100) | parallel -j1 --lb cat
# Pass the data to STDIN, but only after 100 seconds
(echo 1; echo 2; sleep 100) | parallel -j1 --lb --pipe cat

Im using parallel 20190422 on Arch Linux

Upvotes: 3

Views: 1780

Answers (1)

Ole Tange
Ole Tange

Reputation: 33685

# Pass the data to STDIN, but only after 100 seconds
(echo 1; echo 2; sleep 100) | parallel -j1 --lb --pipe cat

This is due to GNU Parallel reads 1 MB by default. So GNU Parallel waits for more input. Only after 100 sec is the STDIN closed, and GNU Parallel gets an EOF.

You can probably do something like this:

(echo 1; echo 2; echo 3; sleep 100) | parallel -j1 --block 1 -N1 --lb --pipe 'date;cat'

But if the lines are much longer, then increase --block.

Upvotes: 1

Related Questions