Turn off buffering

Question

Where is the buffer in this following ... and how do I turn it off?

I am writing out to stdout in a python program like so:

for line in sys.stdin:
    print line

There is some buffering going on here:

tail -f data.txt | grep -e APL | python -u Interpret.py

I tried the following to shake off possible buffering ... with no luck:

• as above using the -u flag with python invocation
• calling sys.stdout.flush() after each sys.stdout.write() call ... all of these create a buffered stream with python waiting something like a minute to print out the first few lines.

• used the following modified command:

  stdbuf -o0 tail -f data.txt | stdbuf -o0 -i0 grep -e APL | stdbuf -i0 -o0 python -u Interpret.py

To benchmark my expectations, I tried:

tail -f data.txt | grep -e APL 

This produces a steady flow of lines ... it surely is not as buffered as the python command.

So, how do I turn off buffering? ANSWER: It turns out there is buffering on both ends of the pipe.

Answer 1

file.readlines() and for line in file have internal buffering which is not affected by -u option (see -u option note). Use

while True:
   l=sys.stdin.readline()
   sys.stdout.write(l)

instead.

By the way, sys.stdout is line-buffered by default if it points to terminal and sys.stderr is unbuffered (see stdio buffering).

Answer 2

The problem is in your for loop. It will wait for EOF before continuing on. You can fix it with a code like this.

while True:
    try:
        line = sys.stdin.readline()
    except KeyboardInterrupt:
        break 

    if not line:
        break

    print line,

Try this out.

Answer 3

The problem, I believe is in grep buffering its output. It is doing that when you pipe tail -f | grep ... | some_other_prog. To get grep to flush once per line, use the --line-buffered option:

% tail -f data.txt | grep -e APL --line-buffered | test.py
APL

APL

APL

where test.py is:

import sys
for line in sys.stdin:
    print(line)

(Tested on linux, gnome-terminal.)

Answer 4

sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0) and make sure PYTHONUNBUFFERED is set in your environment.