CODEWITHSUNDEEP
.netperformanceparsingfile-iobinaryfiles
michael
michael

Reputation: 2597

Is there a tried and true way of accessing specific areas of a very large (>1 GB) binary file very fast in .NET

Using a very large raw binary data file, I will be creating a large binary file (greater than 1 GB, usually greater than 4 GB) that contains sensor data from an arbituary number of sensors (1 to 64). The new processed file will be used by user in a GUI where they will be jumping around the file looking at different time periods or other index that may be defined. It's important that access is as smooth as possible, and I also would like to only load as much as needed (what the user sees plus maybe a bit of a buffer).

I would like to construct the processed data file such that it has an index at the beginning that is then used to jump to whatever point the person want to go.

Should I be writing this from scratch? Or are there some helper libraries that may come in use?

The reason I wouldn't just tack some indexing onto the original raw data file is that it has been created in a way that is definitely not optimal for reading individual sensors (running compression, data stored in DCT blocks, etc...).

Upvotes: 4

Views: 197

Answers (1)

Tergiver
Tergiver

Reputation: 14517

You can use a Memory Mapped File. In .NET 4.0 there are now wrappers: http://msdn.microsoft.com/en-us/library/system.io.memorymappedfiles.memorymappedfile.aspx

If you need to target an earlier version you will have to p/invoke the API methods yourself: http://msdn.microsoft.com/en-us/library/aa366537.aspx

Upvotes: 3

Related Questions