Redirecting tail output into a program

Question

I want to send a program the most recent lines from a text file using tail as stdin.

First, I echo to the program some input that will be the same every time, then send in tail input from an inputfile which should first be processed through sed. The following is the command line that I expect to work. But when the program runs it only receives the echo input, not the tail input.

(echo "new" && tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex' && cat) | ./program

However, the following works exactly as expected, printing everything out to the terminal:

echo "new" && tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex' && cat

So I tried with another type of output, and again while the echoed text posted, the tail text does not appear anywhere:

(echo "new"  && tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex') | tee out.txt 

This made me think it is a problem with buffering, but I tried the unbuffer program and all other advice here (https://superuser.com/questions/59497/writing-tail-f-output-to-another-file) without results. Where is the tail output going and how can I get it to go into my program as expected?

Answer 1

One thing you may be getting confused by -- | (pipeline) is higher precedence than && (consecutive execution). So when you say

(echo "new" && tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex' && cat) | ./program

that is equivalent to

(echo "new" && (tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex') && cat) | ./program

So the cat isn't really doing anything, and the sed output is probably buffered a bit. You can try using the -u option to sed to get it to use unbuffered output:

(echo "new" && (tail -f ~/inputfile 2> /dev/null | sed -n -u -r 'some regex')) | ./program

I believe some versions of sed default to -u when the output is a terminal and not when it is a pipe, so that may be the source of the difference you're seeing.

Answer 2

The buffering problem was resolved when I prefixed the sed command with the following:

stdbuf -i0 -o0 -e0 

Much more preferable to using unbuffer, which didn't even work for me. Dave M's suggestion of using sed's relatively new -u also seems to do the trick.

Answer 3

You can use the i command in sed (see the command list in the manpage for details) to do the inserting at the beginning:

tail -f inputfile | sed -e '1inew file' -e 's/this/that/' | ./program