CODEWITHSUNDEEP
pipegnu-parallel
Stephen Chu
Stephen Chu

Reputation: 373

GNU parallel loses pv progress output

It seems if I use GNU Parallel to wrap a command that uses the pv (pipe viewer) command, it loses the command's progress output.

For example, if I run the following pv command, which shows the progress of my gzip:

› dd if=/dev/urandom bs=256m count=1 | (pv --progress -s 256m --bytes | gzip > /dev/null)
 208MiB [=============================>                                                    ] 32%

... it works fine. However, if I wrap the same command with GNU parallel:

parallel 'dd if=/dev/urandom bs={} count=1 | (pv --progress -s {} --bytes | gzip > /dev/null)' ::: 256m

... I no longer see the in progress gzip progress, but only the final script output in the end.

Question: Not sure if it's my pv or parallel being the problem, but is there a way to show individual command (Unix pipeline) progress (not job progress which parallel --progress gives) if I am using pv?

Upvotes: 3

Views: 541

Answers (1)

Ole Tange
Ole Tange

Reputation: 33685

GNU Parallel saves output to temporary files and when the job is done, it prints these. So it is basically running this for every command:

(cmd; cmd) > tmp.out 2> tmp.err
cat tmp.out
cat tmp.err >&2

This is the reason you do not see any output.

You can ask GNU Parallel to not use temporary files, but only buffer full lines with --line-buffer. That does not work here, because pv really wants a tty.

Luckily you can also ask GNU Parallel to connect the tty with --tty. GNU Parallel assumes that when you connect a tty that you only want a single job to be running (i.e. --jobs 1), but you can override this:

parallel -j2 --tty 'dd  if=/dev/urandom bs=1 count={} | (pv --progress -s {} --bytes | gzip > /dev/null)' ::: 20M 10M

Upvotes: 2

Related Questions