CODEWITHSUNDEEP
matlabbigdata
Arnaldo Batista
Arnaldo Batista

Reputation: 23

Reading variables from big mat files

We have MATLAB mat files whose size is around 12 GB each.

These mat files are MATLAB tables. We only want to read some rows in the table, however we have to endure with loading times of about 10 minutes for a 12 GB mat file. Since we only want to read some rows in the file is it possible to do that without having to load it to MATLAB.

The matfile command will load the full file into memory:

m=matfile('my_big_fat_file.mat')

thus taking the same time to lad as the load command itself.

What, apparently matfile is good is to save individual variables rapidly in the my_big_fat-file using:

m = matfile('my_big_fat_file.mat','Writable',true);

My problem is loading the whole file to read only a couple of variables.

On the other hand the MATLAB datastore command seems to be designed to handle data in csv or image format.

many thanks

Upvotes: 2

Views: 1125

Answers (1)

ibezito
ibezito

Reputation: 5822

Using matfile object enables accessing to variables directly without loading the entire file from memory. It is important to use -7.3 flag when saving the file to make the partial variable reading efficient.

Reading a single line of a variable from mat file

Given a matfile object matfileObj with variable b, you can read the i'th row of b as follows:

matFileObj = matfile('mFile.mat')
matFileObj.b(1,:);

Converting a mat file to v7.3

Converting an existing mat file to efficient one can be done as follows:

load('m.mat');
save('m_v7_3.mat','-v7.3');

Full code example

%generating a mat file which contains 4 variables
a=1;
b=rand(3000,3000);
b(1,:) = 0; 
c=rand(3000,3000);
d=rand(3000,3000);
save('mFile.mat','-v7.3');

%acessing variables without loading the enitre structure 
tic
matFileObj = matfile('mFile.mat')
matFileObj.a;
matFileObj.b(i,:);
toc

%acessing variables regularly
tic 
S = load('mFile.mat');
S.a;
S.b(1,:);
toc

Result

Elapsed time is 0.026882 seconds.
Elapsed time is 8.181252 seconds.

Upvotes: 3

Related Questions