CODEWITHSUNDEEP
pythoninputbinaryfiles
Lin
Lin

Reputation: 5435

Most efficient way of loading formatted binary files in Python

I have binary files no larger than 20Mb in size that have a header section and then a data section containing sequences of uchars. I have Numpy, SciPy, etc. and each library has different ways of loading in the data. Any suggestions for the most efficient methods I should use?

Upvotes: 6

Views: 479

Answers (4)

Toni Ruža
Toni Ruža

Reputation: 7512

bdec seems promising.

Upvotes: 1

Theran
Theran

Reputation: 3866

struct should work for the header section, while numpy's memmap would be efficient for the data section if you are going to manipulate it in numpy anyways. There's no need to stress out about being inconsistent here. Both methods are compatible, just use the right tool for each job.

Upvotes: 4

Eli Bendersky
Eli Bendersky

Reputation: 273816

I found that array.fromfile is the fastest methods for homogeneous data.

Upvotes: 0

rmmh
rmmh

Reputation: 7095

Use the struct module, or possibly a custom module written in C if performance is critical.

Upvotes: 8

Related Questions