CODEWITHSUNDEEP
pythonfile-io
tlnagy
tlnagy

Reputation: 3474

Quick write to a file for a Python subprocess

I have a shell program that needs to be pointed at a text file. I would like to write to a temporary file, run this program on it via Python's subprocess method, and save the output. Here's what I have so far:

with open('test.txt', 'w') as f:
    for i in range(50):
        f.write('sometext %d'%i)
        f.flush()
        output = subprocess.check_output('./program test.txt', shell=True, stderr=subprocess.STDOUT)

Output only gets updated once. I'm assuming that flush isn't able to write to the file before the next iteration of the loop executes. What other ways could this be implemented? Opening and closing the file seems like it would be expensive, because this code is going to be called many times.

Upvotes: 0

Views: 175

Answers (4)

mhawke
mhawke

Reputation: 87084

I'm pretty sure that you realise that output is rebound on each iteration, and will therefore contain only the final value at the end, so I've tried a slightly modified version of your code and found that it works as expected.

output = []
with open('test.txt', 'w') as f:
    for i in range(20):
        f.write('%d ' % i)
        f.flush()
        output.append(subprocess.check_output('cat test.txt', shell=True, stderr=subprocess.STDOUT))

>>> from pprint import pprint
>>> pprint(output)
[b'0 ',
 b'0 1 ',
 b'0 1 2 ',
 b'0 1 2 3 ',
 b'0 1 2 3 4 ',
 b'0 1 2 3 4 5 ',
 b'0 1 2 3 4 5 6 ',
 b'0 1 2 3 4 5 6 7 ',
 b'0 1 2 3 4 5 6 7 8 ',
 b'0 1 2 3 4 5 6 7 8 9 ',
 b'0 1 2 3 4 5 6 7 8 9 10 ',
 b'0 1 2 3 4 5 6 7 8 9 10 11 ',
 b'0 1 2 3 4 5 6 7 8 9 10 11 12 ',
 b'0 1 2 3 4 5 6 7 8 9 10 11 12 13 ',
 b'0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 ',
 b'0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 ',
 b'0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 ',
 b'0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 ',
 b'0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 ',
 b'0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 ']

Upvotes: 0

glglgl
glglgl

Reputation: 91049

If you want to omit closing and re-opening (which would probably be the cleanest way), you might want to f.seek(0) at the start and f.truncate() after writing.

Otherwise, the data is appended to the file on every loop run and the external program might be confused.

Upvotes: 1

Mattias Backman
Mattias Backman

Reputation: 955

You may need to call os.fsync() as well to make sure that the changes are not just flushed but forcefully written to disk as well.

Upvotes: 0

soupault
soupault

Reputation: 6510

f.flush will not definitely write to disk, if buffer isn't filled.

Check this

Upvotes: 0

Related Questions