CODEWITHSUNDEEP
linuxbashpipe
Duke Leto
Duke Leto

Reputation: 215

How to slow down first program in pipe until second program is ready

I have a script running as so:

cat in | prog1 2>log1 | prog2 2>log2 | prog3 2>log3 > out

<in> can have millions of lines and I want each program to only produce output on STDOUT when the next program's STDIN has been consumed.

This is because prog1 uses system resources that prog2 cleans up once it's done with it.

So basically I want cat to "pause" between line writes until said lines have been consumed by prog1, and onward down the line.

I have tried using stdbuf but that doesn't appear to have any effect, i.e.:

stdbuf -o0 cat in | stdbuf -o0 -i0 prog1 2>log1 | stdbuf -o0 -i0 prog2 2>log2 | stdbuf -o0 i0 prog3 2>log3 > out

Using pv -l between each pipe confirms the speed difference, and also tail -f of the different log files.

Is there a way to do this?

I am aware that this will adversely affect performance.

PS I also want the whole pipe to exit when any of the programs exit, and each of the programs are legacy written in either perl or python, so they can't just be combined.

Thanks in advance!

Upvotes: 0

Views: 287

Answers (1)

vintnes
vintnes

Reputation: 2030

Why do you insist on using pipes? The whole point of piping data is "First In, First Out". If you want regimented processes, use temporary files.

prog1 < in 2>log1 > $(mktemp)
prog2 < $_ 2>log2 > $(mktemp)
# etc...

edit: In this context, $_ means "the last argument of the last command line".

Upvotes: 1

Related Questions