Improve speed of reading and converting from binary file?

Question

I know there have been some questions regarding file reading, binary data handling and integer conversion using struct before, so I come here to ask about a piece of code I have that I think is taking too much time to run. The file being read is a multichannel datasample recording (short integers), with intercalated intervals of data (hence the nested for statements). The code is as follows:

# channel_content is a dictionary, channel_content[channel]['nsamples'] is a string
for rec in xrange(number_of_intervals)):
    for channel in channel_names:
        channel_content[channel]['recording'].extend(
            [struct.unpack( "h", f.read(2))[0]
            for iteration in xrange(int(channel_content[channel]['nsamples']))])

With this code, I get 2.2 seconds per megabyte read with a dual-core with 2Mb RAM, and my files typically have 20+ Mb, which gives some very annoying delay (specially considering another benchmark shareware program I am trying to mirror loads the file WAY faster).

What I would like to know:

1. If there is some violation of "good practice": bad-arranged loops, repetitive operations that take longer than necessary, use of inefficient container types (dictionaries?), etc.
2. If this reading speed is normal, or normal to Python, and if reading speed
3. If creating a C++ compiled extension would be likely to improve performance, and if it would be a recommended approach.
4. (of course) If anyone suggests some modification to this code, preferrably based on previous experience with similar operations.

Thanks for reading

(I have already posted a few questions about this job of mine, I hope they are all conceptually unrelated, and I also hope not being too repetitive.)

Edit: channel_names is a list, so I made the correction suggested by @eumiro (remove typoed brackets)

Edit: I am currently going with Sebastian's suggestion of using array with fromfile() method, and will soon put the final code here. Besides, every contibution has been very useful to me, and I very gladly thank everyone who kindly answered.

Final Form after going with array.fromfile() once, and then alternately extending one array for each channel via slicing the big array:

fullsamples = array('h')
fullsamples.fromfile(f, os.path.getsize(f.filename)/fullsamples.itemsize - f.tell())
position = 0
for rec in xrange(int(self.header['nrecs'])):
    for channel in self.channel_labels:
        samples = int(self.channel_content[channel]['nsamples'])
        self.channel_content[channel]['recording'].extend(
                                                fullsamples[position:position+samples])
        position += samples

The speed improvement was very impressive over reading the file a bit at a time, or using struct in any form.

Answer 1

You could use array to read your data:

import array
import os

fn = 'data.bin'
a = array.array('h')
a.fromfile(open(fn, 'rb'), os.path.getsize(fn) // a.itemsize)

It is 40x times faster than struct.unpack from @samplebias's answer.

Answer 2

A single array fromfile call is definitively fastest, but wont work if the dataseries is interleaved with other value types.

In such cases, another big speedincrease that can be combined with the previous struct answers, is that instead of calling the unpack function multiple times, precompile a struct.Struct object with the format for each chunk. From the docs:

Creating a Struct object once and calling its methods is more efficient than calling the struct functions with the same format since the format string only needs to be compiled once.

So for instance, if you wanted to unpack 1000 interleaved shorts and floats at a time, you could write:

chunksize = 1000
structobj = struct.Struct("hf" * chunksize)
while True:
    chunkdata = structobj.unpack(fileobj.read(structobj.size))

(Note that the example is only partial and needs to account for changing the chunksize at the end of the file and breaking the while loop.)

Answer 3

Not sure if it would be faster, but I would try to decode chunks of words instead of one word a time. For example, you could read 100 bytes of data a time like:

s = f.read(100)
struct.unpack(str(len(s)/2)+"h", s)

Answer 4

extend() acepts iterables, that is to say instead of .extend([...]) , you can write .extend(...) . It is likely to speed up the program because extend() will process on a generator , no more on a built list

There is an incoherence in your code: you define first channel_content = {} , and after that you perform channel_content[channel]['recording'].extend(...) that needs the preliminary existence of a key channel and a subkey 'recording' with a list as a value to be able to extend to something

What is the nature of self.channel_content[channel]['nsamples'] so that it can be submitted to int() function ?

Where do number_of_intervals come from ? What is the nature of the intervals ?

In the rec in xrange(number_of_intervals)): loop , I don't see anymore rec . So it seems to me that you are repeating the same loop process for channel in channel_names: as many times as the number expressed by number_of_intervals . Are there number_of_intervals * int(self.channel_content[channel]['nsamples']) * 2 values to read in f ?

I read in the doc:

class struct.Struct(format)

Return a new Struct object which writes and reads binary data according to the format string format. Creating a Struct object once and calling its methods is more efficient than calling the struct functions with the same format since the format string only needs to be compiled once.

This expresses the same idea as samplebias.

If your aim is to create a dictionary, there is also the possibility to use dict() with a generator as argument

.

EDIT

I propose

channel_content = {}
for rec in xrange(number_of_intervals)):
    for channel in channel_names:
        N = int(self.channel_content[channel]['nsamples'])
        upk = str(N)+"h", f.read(2*N)
        channel_content[channel]['recording'].extend(struct.unpack(x) for i,x in enumerate(upk) if not i%2)

I don't know how to take account of the J.F. Sebastian's suggestion to use array

Answer 5

If the files are only 20-30M, why not read the entire file, decode the nums in a single call to unpack and then distribute them among your channels by iterating over the array:

data = open('data.bin', 'rb').read()
values = struct.unpack('%dh' % len(data)/2, data)
del data
# iterate over channels, and assign from values using indices/slices

A quick test showed this resulted in a 10x speedup over struct.unpack('h', f.read(2)) on a 20M file.