CODEWITHSUNDEEP
pythonsubprocesspopen
plok
plok

Reputation: 706

Unbuffered subprocess output (last line missing)

I must be overlooking something terribly obvious. I need to execute a C program, display its output in real time and finally parse its last line, which should be straightforward as the last line printed is always the same.

process = subprocess.Popen(args, shell = True,  
                           stdout = subprocess.PIPE, stderr = subprocess.PIPE)

# None indicates that the process hasn't terminated yet.
while process.poll() is None:

    # Always save the last non-emtpy line that was output by the child
    # process, as it will write an empty line when closing its stdout.
    out = process.stdout.readline()
    if out:
        last_non_empty_line = out

    if verbose:
        sys.stdout.write(out)   
        sys.stdout.flush()

# Parse 'out' here...

Once in a while, however, the last line is not printed. The default value for Popens's bufsize is 0, so it is supposed to be unbuffered. I have also tried, to no avail, adding fflush(stdout) to the C code just before exiting, but it seems that there is absolutely no need to flush a stream before exiting a program.

Ideas anyone?

Upvotes: 3

Views: 2871

Answers (2)

Douglas Leeder
Douglas Leeder

Reputation: 53310

readline() has to buffer the text, waiting for a new-line.

You'll always have a race condition - no amount of un-buffered streams will handle the fact that you're handling a line, then checking for exit, then reading a line, so if the subprocess exits while you're handling a line, you won't read anything else. In addition the shell is probably introducing its own buffering.

So you could either:

  1. Use communicate() and give up the verbose output while the subprocess is running.

  2. Make sure you continue reading after the process has exited, until you get EOF.

I would also suggest altering your code so that you don't have to use shell=True.

Upvotes: 2

Wolph
Wolph

Reputation: 80031

The problem is that you are reading lines till the process exits, (process.poll()) while you do use buffering because of the shell flag.

You would have to keep reading process.stdout till you reach the end of the file or the empty line.

Upvotes: 3

Related Questions