CODEWITHSUNDEEP
matlabimportlarge-files
Sherbika
Sherbika

Reputation: 1

Fast way of importing multiple large files in matlab

I need to read about 4000 data files, each about 400 kB. The data will be analysed later so I wrote the files into a data structure. The importing operation takes about 4 mins and I have tried importdata and dlmread too but there is not much difference.

Please let me know if it is the loop, the import function or Matlab is simply slow at importing multiple large files. catch/try is used because some of these files can't be read properly but it doesn't seem to be slowing the script down.

Here's the script: 

for k=40020:10:75000
try
    name=['tmp' sprintf('%d',k)];
    c=c+1;
    m(k).count=k;
    m(k).col=load(name);
    [val in]=find(m(k).col(:,5)~=1);
    m(k).id=m(k).col(val,1);
    m(k).posx=m(k).col(val,2);
    m(k).posy=m(k).col(val,3);
    m(k).posz=m(k).col(val,4);
catch
    disp(['Error'])
end
end

Upvotes: 0

Views: 537

Answers (1)

matlabgui
matlabgui

Reputation: 5672

A few things to note:

400 kB is not a large file.

4000 files in 4 minutes is 0.06 seconds each.

You dont appear to use the variable c.

Your matrix index starts at 40020 and each loop the next struct index containing data is +10 etc.... This is very sparse which is a waste of memory and a small amount of time.

You state you use dlmread and import data -> but then in the code you are using load. Is the files ascii?

The best way to find out where the time is taken is to use the profiler. 

  profile on
  % run your code
  profile viewer

Are the files local or on a network? Reading files from a network performance can be a lot slower.

Upvotes: 1

Related Questions