Cannot successfully redirect stdout from Popen to temp file

Question

The title says it all, really. Currently running Python 2.7.8, using subprocess32. Minimum reproducing snippet, copied directly from the REPL:

Python 2.7.8 (default, Aug 14 2014, 13:26:38)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import tempfile
>>> import subprocess32 as subp
>>> tfil = tempfile.TemporaryFile()
>>> tfil
<open file '<fdopen>', mode 'w+b' at 0x7f79dcf158a0>
>>> proc = subp.Popen(['ls', '-al'], stdout=tfil, stderr=subp.STDOUT)
>>> proc
<subprocess32.Popen object at 0x7f79dcf04a50>
>>> proc.wait()
0
>>> tfil.read()
''
>>>

Changes to the code that made no difference in the result, in any combination:

• Using subprocess instead of subprocess32
• Using NamedTemporaryFile instead of TemporaryFile
• Setting bufsize=0 on either or both of tfil and proc
• If I try subp.Popen(['ls', '-al'], stdout=sys.stdout, stderr=subp.STDOUT), the output redirects to the terminal just fine - but I need to capture it somewhere else, I can't just mix it together with actual terminal output. Redirecting sys.stdout to the temp file, and then setting it back after the process is done, doesn't work either.

I need to do this because the command I want to use in the actual non-minimal-example program (qacct -j) has an output that's too big for subprocess.PIPE, raising a MemoryError (no further details).

The most relevant existing question I've found is this, but that didn't actually help.

Answer 1

The file is properly written to, but your tfil.read() command will start reading at the point where the last write happened, so it will return nothing. You need to call tfil.seek(0) first.