CODEWITHSUNDEEP
pythonfile-iobinary
Jim Hunziker
Jim Hunziker

Reputation: 15370

How can I efficiently pack binary data into a file in Python?

I would like to write binary data to a file in Python without sending it through temporary buffers first. How can I use the struct module directly on a file?

Upvotes: 0

Views: 977

Answers (1)

Jim Hunziker
Jim Hunziker

Reputation: 15370

Here is how I was able to pack binary data directly into a file under Python 3. The only drawback is that you need to guess a maximum size for the file before writing the data. (An additional call to truncate can be made at the end if the guess is too big.)

Two things are going on here. The file is being memory mapped, and struct is being used to pack data into a memoryview of that memory map. By using a memoryview, it is possible to use the Python buffer interface to write directly to the file. struct's pack_into function can write into anything that supports the buffer interface. This technique can also be used by using a memoryview on a socket to write binary data directly to a socket.

import struct
import mmap

with open('test.bin', 'wb') as f:
    f.truncate(100)

with open('test.bin', 'r+b') as f:
    m = mmap.mmap(f.fileno(), 0)
    mv = memoryview(m)
    for ind in range(25):
        struct.pack_into('>l', mv, ind * 4, ind)

Also, note that it's probably better to make fewer calls to pack_into, and calling it in a loop here is just for illustrative purposes.

Upvotes: 3

Related Questions