CODEWITHSUNDEEP
matlabout-of-memory
shi ma
shi ma

Reputation: 51

A solution for "out of memory" error in matlab

i have a very big size of text file(about 11GB) that needs to load in matlab.but when i use "textread" function,"out of memory" error occurs.and There is no way to reduce the file size. when i type memory, show this to me.

memory
Maximum possible array:     24000 MB (2.517e+10 bytes) *
Memory available for all arrays:     24000 MB (2.517e+10 bytes) *
Memory used by MATLAB:      1113 MB (1.167e+09 bytes)
Physical Memory (RAM):     16065 MB (1.684e+10 bytes)

*  Limited by System Memory (physical + swap file) available.

Does anyone have a solution to this problem?

Upvotes: 2

Views: 618

Answers (2)

Edric
Edric

Reputation: 25160

@Anthony suggested a way to read the file line-by-line, which is perfectly fine, but more recent (>=R2014b) versions of MATLAB have datastore functionality, which is designed for processing large data files in chunks.

There are several types of datastore available depending on the format of your text file. In the simplest cases (e.g. CSV files), the automatic detection works well and you can simply say

ds = datastore('myCsvFile.csv');
while hasdata(ds)
    chunkOfData = read(ds);
    ... compute with chunkOfData ...
end

In even more recent (>=R2016b) versions of MATLAB, you can go one step further and wrap your datastore into a tall array. tall arrays let you operate on data that is too large to fit into memory all at once. (Behind the scenes, tall arrays perform computations in chunks, and give you the results only when you ask for them via a call to gather). For example:

tt = tall(datastore('myCsvFile.csv'));
data = tt.SomeVariable;
result = gather(mean(data)); % Trigger tall array evaluation

Upvotes: 5

Anthony
Anthony

Reputation: 3801

According to your clarification of the purpose of your code:

it is a point cloud with XYZRGB column in txt file and i needs to add another column to this.

What I suggest you to do is read the text file one line at a time, modify the line and write the modified line straight to a new text file.

To read one line at a time:

% Open file for reading.
fid = fopen(filename, 'r');

% Get the first line.
line = fgetl(fid);
while ~isnumeric(line)
    % Do something.

    % get the next line
    line = fgetl(fid);
end
fclose(fid);

To write the line, you can use fprintf.

Here is a demonstration:

filename = 'myfile.txt';
filename_new = 'myfile_new.txt';

fid = fopen(filename);
fid_new = fopen(filename_new,'w+');
line = fgetl(fid);
while ~isnumeric(line)
    % Make sure you add \r\n at the end of the string; 
    % otherwise, your text file will become a one liner.
    fprintf(fid_new, '%s %s\r\n', line, 'new column');
    line = fgetl(fid);
end
fclose(fid);
fclose(fid_new);

Upvotes: 2

Related Questions