CODEWITHSUNDEEP
matlabmemorytext-fileslarge-files
Prefoninsane
Prefoninsane

Reputation: 103

Importing large amount of data into MATLAB?

I have a text file that is ~80MB. It has 2 cols and around 6e6 rows. I would like to import the data into MATLAB, but it is too much data to do with the load function. I have been playing around with the fopen function but cant get anything to work.

Ideally I would like to take the first col of data and import and eventually have it in one large array in MATLAB. If that isn't possible, I would like to split it into arrays of 34,013 in length. I would also like to do the same for the 2nd col of data.

Upvotes: 0

Views: 174

Answers (2)

lakshmen
lakshmen

Reputation: 29064

fileID = fopen('yourfilename.txt');
formatSpec = '%f %f';
while ~feof(fileID)
 C = textscan(fileID,formatSpec,34013);
end

Hope this helps..

Edit:

The reason you are getting error is because C has two columns. So you need to take the columns individually and handle them.

For example:

column1data = reshape(C(:,1),301,113);
column2data = reshape(C(:,2),301,113);

Upvotes: 2

anandr
anandr

Reputation: 1662

You may also consider to convert your file to binary format if your data file is not changing each time you want to load it. Then you'll load it way much faster. Or you may do "transparent binary conversion" like in the function below. Only first time you load the data will be slow. All subsequent will be fast.

function Data = ReadTextFile(FileName,NColumns)
    MatFileName = sprintf('%s.mat',FileName);   % binary file name
    if exist(MatFileName,'file')==2             % if it exists
        S = load(MatFileName,'Data');           % load it instead of
        Data = S.Data;                          % the original text file
        return;
    end
    fh        = fopen(FileName);                % if binary file does not exist load data ftom the original text file
    fh_closer = onCleanup( @() fclose(fh) );    % the file will be closed properly even in case of error
    Data      = fscanf(fh, repmat('%f ',1,NColumns), [NColumns,inf]);
    Data      = Data';
    save(MatFileName,'Data');                   % and make binary "chache" of the original data for faster subsequent reading
end

Do not forget to remove the MAT file when the original data file is changed.

Upvotes: 0

Related Questions