CODEWITHSUNDEEP
pythonpython-3.xstdout
Andrew Pinkham
Andrew Pinkham

Reputation: 630

Redirecting `sys.stdout` to a file or a buffer

I am working in Python 3.4, and I have behavior that I don't understand: if I redirect stdout to a file, I am able to capture text from child processes. However, when I redirect to a Python file object, I stop capturing that output. I would love an explanation of the (following) behavior.

I have:

from multiprocessing import Process

def worker():
    print('forked output')

def output():
    print('redirected')
    p = Process(target=worker)
    p.daemon = True
    p.start()
    p.join()  # wait for subprocess to terminate
    print('end')

The redirect_stdout context manager in Python 3.4 makes grabbing stdout easy (in this instance).

from contextlib import redirect_stdout
from sys import stdout
from tempfile import TemporaryFile


with TemporaryFile(mode='w+', encoding=stdout.encoding) as buf:
    with redirect_stdout(buf):
        output()  # the function defined above
    buf.seek(0)
    s = buf.read()
    print('output from TemporaryFile:')
    print(s)

I can then simply call the script to get the following output:

$ python stackoverflow.py 
output from TemporaryFile:
redirected
forked output
end

This is exactly what I want, and works fine.

My confusion stems from the fact that if I if I switch TemporaryFile with TextIOWrapper, the behavior of my script changes.

from io import BytesIO, TextIOWrapper


with TextIOWrapper(BytesIO(), stdout.encoding) as buf:
    with redirect_stdout(buf):
        output()  # the function defined at the start
    buf.seek(0)
    s = buf.read()
    print('output from TextIO:')
    print(s)

Now when I call the program, I lose the output from the forked process.

$ python stackoverflow.py 
output from TextIO:
redirected
end

What is going on?

I suspect the problem has to do with the fact that the TextIOWrapper object doesn't have a file descriptor, and that os.fork() (used by multiprocessing) may thus be replacing the TextIOWrapper with another, but I admit some confusion there (especially given that stdout appears to be a TextIOWrapper with fileno() implemented).

>>> from sys import stdout
>>> stdout.fileno()
1
>>> stdout
<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>

Thanks for any information.

Upvotes: 8

Views: 1114

Answers (1)

Kevin
Kevin

Reputation: 30151

Since you're using multiprocessing, you should use the standard message passing primitives provided by that library. Do not call print() from the subprocess; that's poor design.

If you're actually trying to make this work with someone else's (non-Python) code, use subprocess.check_output() or another of the subprocess functions instead.

Upvotes: 1

Related Questions