CODEWITHSUNDEEP
pythonbinarybytebuffer
user3157868
user3157868

Reputation: 61

MemoryError while writing large binary file

All of the related questions i researched involved reading from a file and writing to another. I am generating a large file from scratch. I'll simplify it here:

import os

fo = open("example.example", "wb", 0)

fo.write ((b'\x01\x01\x01\x01')+(((b'\xff') * 1074000000)+ (b'\x01\x01\x01\x01;')))

fo.close()

This example should create a file a little larger that a gig... I know I need to use buffer during writing process, but not sure how when simply writing from scratch in bytes. THANKS IN ADVANCE

Upvotes: 2

Views: 4675

Answers (1)

Jan Vlcinsky
Jan Vlcinsky

Reputation: 44092

You are asking Python to create more than 1074000000 bytes in memory. This logically causes MemoryError.

import os

with open("example.example", "wb") as f:
    f.write(b'\x01\x01\x01\x01')
    for i in xrange(1074000000):
        f.write(b'\xff')
    f.write(b'\x01\x01\x01\x01;')

Of course, you can write the data in the loop in bigger chunks to make it more effective, but now you shall have the idea.

In case you use Python 3, replace xrange with range.

Using range in Python 2 could be a problem, as it would attempt to create an array of 1074000000 integers in memory. xrange instead creates a generator, which is returning the numbers one by one, but not recreating them in advance. This has became standard practice in Pyton 3 and xrange was removed in favour of range.

Upvotes: 1

Related Questions