CODEWITHSUNDEEP
pythonfilebufferflush
crypto5
crypto5

Reputation: 851

Writing to file doesn't flush content automatically and cause out of memory in Python

I made simple python program to generate big text file:

import sys
import random

f = open('data.txt', 'w')
for i in range(100000000):
        f.write(str(i) + "\t" + str(random.randint(0,1000)) + "\n")
f.close()

When I launch it using CPython it eat all available OS memory and write nothing to the file.

When I launch it on Jython I get OutOfMemoryException.

As far as I understand it stores everything in memory buffer and never did flush before close() call.

My question is: how to limit the file buffer and trigger autoflush? I don't want to callflush() manually, I think it's wrong from performance point of view. I want flush() to be called automatically when file buffer is overloaded or so.

Thanks!

Upvotes: 1

Views: 755

Answers (2)

mob
mob

Reputation: 118665

Buffering is not the problem. The problem is calling the range() function with a giant argument, which will attempt to allocate an array with lots of elements. You will get the same error if you just say

r = range(100000000)

Upvotes: 9

David Berger
David Berger

Reputation: 12803

Have you tried passing in a buffer size to the open function?

f = open('data.txt', 'w', 5000)

Upvotes: -1

Related Questions