MATLAB: How can I efficiently read in these data files?

Question

I have 100 data files in a folder called "Experiment1", and I need to take all of the data from them and put them into a single matrix. Each data file contains 15 columns and 40 rows of data.

The order in which the files are in the folder is arbitrary. It doesn't matter in what order they get put into the combined matrix.

I've written some code using dlmread that will do the job:

for i = 1:100

    %% Read in the relevant file.
    filename = ['File_' int2str(i) '.dat']
    Data = dlmread(fullfile(pwd, 'Experiment1',filename));

    %% Put all the data I need in a separate matrix
    NeededData(1+((i-1)*40):i+((i-1)*40)-i+40,1:15) = Data(:,1:15);

end

However, there are two things I don't like about my code.

1. The files have random names at present, and I'd need to manually change all their names to "File_1.dat", "File_2.dat", etc.

2. The code is cumbersome and hard to read.

How could I do things better?

Answer 1

Since you've fixed the problem of defining the name of the files to be read with dir, you can improve the way you add the read data (Data) to the output matrix (NeededData).

You can sumultaneously read the input files and add the data to the output matrix by inserting the call to dlmread directly in the assignment statement:

files=dir('*.dat');
n_files=length(files)
% Initialize the output matrix as empty
NeededData_0=[]
for i=1:n_files
   % Simultaneously read input file and assign data to the output matrinx
   NeededData_0=[NeededData_0;dlmread(files(i).name)]
end

In case you prefer working with the inides (as in your origina approach), since you know in advance that all the files have the same (40) number of rows) you can simplify the notation as follows:

files=dir('*.dat');
n_files=length(files)
% Define the number of rows in each inout file
n_rows=40;
% Define the number of colums in each inout file
n_col=15;
NeededData_2=nan(n_rows*n_files,n_col)
% Define the sequence of rows
r_list=1:n_rows:n_rows*n_files
for i=1:3
   Data=dlmread(files(i).name)
   NeededData_2(r_list(i):r_list(i)+n_rows-1,:)=Data
end

Hope this helps.

Answer 2

Using the suggestion to use dir present in the answers I have made the following code, which is clearly an improvement on my earlier effort. I would welcome further improvements, or suggestions for alternative approaches.

files = dir('*.dat');

for i = 1:length({files.name})

    %% Read in the relevant file.
    Data = dlmread(files(i).name);

    %% Put all the data I need in a separate matrix
    NeededData(1+((i-1)*40):i+((i-1)*40)-i+40,1:15) = Data(:,1:15);

end