CODEWITHSUNDEEP
matlabmemoryoptimizationiofread
Dev-iL
Dev-iL

Reputation: 24169

Working with binary files in MATLAB: load as-is to memory and interpret later

I have some binary files (obtained from a camera), in which frames are stored in some sequential order.

The usual algorithm to working with such files would be something along the lines of: open file -> advance to first frame -> read frame -> advance a few bits until the next frame -> read next frame and so on. Intuitively, this results in many "short" I\O operations that likely reduce performance when compared with a single "long" operation.

For this reason, it seemed natural to me to initially load the file into memory, and interpret it later "at my leisure" (since reading from RAM is supposed to be much faster than from disk).

So far, I could think of two possible solutions to this problem:

  1. The submission mfile in the FileExchange. {Even though it looks exactly like what I need, I'm slightly reluctant to use this solution because it's about 7 years old (which invites the assumption that it may be outdated)}
  2. Using a "Ram Disk" to hold the files, and then use the standard MATLAB I\O tools.

My question is: Is there some built-in functionality in MATLAB that allows to achieve this result?

Upvotes: 0

Views: 427

Answers (1)

Dev-iL
Dev-iL

Reputation: 24169

Following the suggestions of @Ben Voigt and @Peter, I tried writing some code that reads the binary file to memory and parses it. Below is a comparison of a piece of the original code and my "from memory" implementation that produces the same result (this is just the header and not the frames themselves, but the idea is similar +reshape).

I'd like to point out the documentation in fread.m is very concise and informative.

Original Code (w/o comments)

fid=fopen(fName,'r');

fseek(fid, 11, 'bof');
ptw.m_MainHeaderSize=fread(fid,1,'int32');
ptw.m_FrameHeaderSize=fread(fid,1,'int32');

fseek(fid, 27, 'bof');
ptw.m_nframes=fread(fid,1,'int32');

fseek(fid, 212 , 'bof');
ptw.m_fHousingTemp=fread(fid,1,'float');
fseek(fid, 216 , 'bof');
ptw.m_fHousingTemp2=fread(fid,1,'float');

fseek(fid, 245, 'bof');
ptw.m_minlut=fread(fid,1,'int16');
ptw.m_maxlut=fread(fid,1,'int16');

"From-memory" Equivalent

function dts = InterpretFromMemory(fName)
%% Definitions
UINT8_IN_CHAR = 1;
UINT8_IN_INT16 = 2;
UINT8_IN_INT32 = 4; 
UINT8_IN_FLOAT = 4;
ZERO_POS = 1;
%% Initialization
dts = struct('m_filename',fName);
%% Reading
fid=fopen(fName);
fComplete = fread(fid,'*uint8'); 
fclose(fid);
%% Parsing
curr_pos = 11;
dts.m_MainHeaderSize = readFromArrayAndAdvance(fComplete,1,'int32');
dts.m_FrameHeaderSize = readFromArrayAndAdvance(fComplete,1,'int32');

curr_pos = 27;
dts.m_nframes = readFromArrayAndAdvance(fComplete,1,'int32');

curr_pos = 212;
dts.m_fHousingTemp  = readFromArrayAndAdvance(fComplete,1,'float');
dts.m_fHousingTemp2 = readFromArrayAndAdvance(fComplete,1,'float');

curr_pos = 245;
dts.m_minlut = readFromArrayAndAdvance(fComplete,1,'int16');
dts.m_maxlut = readFromArrayAndAdvance(fComplete,1,'int16');

function [out]=readFromArrayAndAdvance(hArray,numVals,valType)
    switch valType
        case 'char'
            numBytes = UINT8_IN_CHAR;
        case {'int16','uint16'}
            numBytes = UINT8_IN_INT16;
        case {'int32','uint32'}
            numBytes = UINT8_IN_INT32;  
        case 'float'
            numBytes = UINT8_IN_FLOAT;
            valType = 'single'; %// float equivalent supported by typecast
    end
    out = typecast(hArray(...
        curr_pos+ZERO_POS:curr_pos+ZERO_POS+numVals*numBytes-1),valType);
    curr_pos = curr_pos + numVals*numBytes;
end

end

TODO: I intend to run some benchmarks on this in the future and compare the various methods.

Copyright notice: the "original code" snippet belongs to FLIR.

Upvotes: 1

Related Questions