CODEWITHSUNDEEP
pythonoutputstdout
rfmoz
rfmoz

Reputation: 1111

Python script differs output between terminal and redirection to a file

I am writing a large python script with various subprocess.call to execute commands available in the system, and I have an issue because the output differs if it is printed into the terminal or if is redirected to a file.

For reproduce the problem, this is a small part of the script:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from subprocess import call

print "Hello at first"
call(["rsync", "-aHvz", "root@server1:/tmp/a", '.'])
print "Hello at the end"

Executing it from terminal return that in the correct order, print + rsync + print:

$ python keepconf.py
Hello at the first
receiving incremental file list

sent 12 bytes  received 123 bytes  270.00 bytes/sec
total size is 55858143  speedup is 413764.02
Hello at the end

Executing the same, but redirecting the output to a file:

$ python keepconf.py  > /tmp/output
$ cat /tmp/output
receiving incremental file list

sent 12 bytes  received 123 bytes  270.00 bytes/sec
total size is 55858143  speedup is 413764.02
Hello at the first
Hello at the end

The order now is rsync + print + print. Why is this behaviour?

Upvotes: 2

Views: 900

Answers (1)

Martijn Pieters
Martijn Pieters

Reputation: 1122012

Output for a terminal (or more precisely, a tty) is typically opened in line buffer mode in Python. When you use a pipe, Python'll use a different buffer, of a fixed size.

This means that when you write text with a newline, the buffer is automatically flushed when printing to a terminal, but to a pipe it'll only be flushed when the buffer is full or a flush is forced (such as when the Python script exits).

In other words, when writing to the terminal the first print line is flushed out to the terminal before the rsync command is run. When redirecting to a pipe, the text is kept in a buffer, the rsync command output is run (writing to the pipe when it flushes, at least once at the end but possibly more often), after which you write some more to the buffer and that buffer is flushed to the pipe when Python exists.

You can force the flush manually:

import sys

# ....
print "Hello at first"
sys.stdout.flush()
call(["rsync", "-aHvz", "root@server1:/tmp/a", '.'])
print "Hello at the end"
sys.stdout.flush()

Upvotes: 4

Related Questions