CODEWITHSUNDEEP
pythonio-redirection
DEV
DEV

Reputation: 2176

Read from a file that my program's output is redirected to

I have a python program. It outputs some lines. I run it like -

python process.py > output.txt

While executing process.py I want to open output.txt from process.py and use the saved data. 

 with open("output.txt", 'r', encoding='utf-8') as template_file:
      template_file_content = template_file.readlines()
            for line in template_file_content:
                # use it

Is there any way to do it.

Upvotes: 0

Views: 56

Answers (4)

ivan_pozdeev
ivan_pozdeev

Reputation: 35998

You are not supposed to do that when you invoke the program the way you did -- python process.py > output.txt. Change the way your program accepts the output file to python process.py output.txt.

With > output.txt, your program writes to sys.stdout. It's not your program's business what happens to this data next -- it's the user's business. That's the interface of the standard streams, and you sign up for this when using them.
Even if you might be able to somehow subvert this limitation, this would be violating the interface (thus making your program problematic) and unreliable (e.g. the output may be not to a file but to any number of other possible destinations depending on the OS).

Conversely, if you accept an output file as an argument, there are no limitations on what you are allowed to do to it while your program is working -- beyond what you yourself guarantee in your program's documentation. E.g. if this is an "output file", you only promise to the user that after your program finishes, this file will have its "output".
However, reading from an "output file" is still uncommon, and you need to explicitly state that you may do this in your program's help -- because that means that the user must ensure that the destination is infinitely readable, too, not just writable (e.g. they may pass it a named pipe otherwise).

Upvotes: 2

tripleee
tripleee

Reputation: 189337

No, the result of an output redirection is not reliably available to the producing process until it finishes. In particular, many OSes will apply output buffering, meaning that the file is only guaranteed to be completely written when the shell closes the open file handle, after Python has finished running.

A possible workaround might be to write a wrapper for process.py and capture its output to a variable and only then write it to standard output.

If you are in a position to modify process.py, change it to return or yield its output to the caller if it doesn't already have a code path which does this, then simply import it from your code instead of running it as a separate process with redirection.

Upvotes: 3

Andreas
Andreas

Reputation: 2521

One way to do it, instead of using the data from output.txt, is to save in a list in the program itself. That way you don't have to deal with the file access conflict.

Another way is to put output.txt as a parameter to your python program and write to it from inside the program. Close the file after you finished writing and open it again when you are reading from it.

Upvotes: 1

Abe Kipnis
Abe Kipnis

Reputation: 25

You might have to save (close) the file first before you try to access the data within it. You might want to try using os.system() calls to 'touch' the file, i.e. create a new file with the name 'output.txt', then write, then read again.

Upvotes: -3

Related Questions