CODEWITHSUNDEEP
matlabmemory
aspirin
aspirin

Reputation: 347

'Out of memory' in Matlab. A slow but a permanent solution?

I am wondering if my suggestion to 'Out of Memory' problem is impossible. Here is my suggestion:

The idea is seamlessly saving huge matrices (say BIG = rand(10^6)) to HDD as a .mat(-v7.3) file when it is not possible to keep it in memory and call it seamlessly whenever required. Then, when you want to use it like:

a = BIG(3678,2222);
s = size(BIG);

, it seamlessly does this behind the scene:

m = matfile('BIG.m');
a = m.BIG(3678,2222);
s = size(m,'BIG');

I know that speed is important but suppose that I have enough time but not enough memory. And also its better to write an memory efficient program but again suppose that I need to use someone else's function which cant be optimized. I do actually have some more related questions: Can this be implemented using objects? Or does it require a infrastructural change in Matlab?

Upvotes: 1

Views: 262

Answers (3)

Andreas H.
Andreas H.

Reputation: 6105

What could probably be a solution to your problem is memory mapped io (which matlab supports).

There a file is mapped to the memory and all read/writes to that memory address are actually read/writes to the file.

This only reserves/blocks memory addresses, it does not consume physical memory.

However, I would only advise it with 64 bit matlab, since with 32 bit matlab the address space is simply not large enough to use ram for data, code for matlab and dlls, and memory mapped io.

Check out the examples for the documentation page of memmapfile(), e.g.,

m = memmapfile('records.dat',    ...
           'Offset', 1024,   ...
           'Format', {'uint32' [4 10 18] 'x'});

A = m.Data(1).x;

whos A
   Name      Size                     Bytes  Class

   A        4x10x18                   2880   uint32 array

Note that, accesses to m.Data(1).x redirect to file IO, i.e. no memory is consumed. So it provides efficient random access to parts of possibly very large data files residing on disk. Also note, that more complex data structures as in the example can be realized.

It seems to me that this provides the "implementation with objects" you had in mind.

Unfortunately, this does not allow to memmap MATfiles directly, which would be really useful. Probably this is difficult because of the compression.

Upvotes: 2

Rody Oldenhuis
Rody Oldenhuis

Reputation: 38042

Seems to me like this is certainly possible, as this essentially what many operating systems do in the form of paging.

Moreover, something similar is provided by MATLAB Distributed Computing Server. This allows you (among other things) to store the data for a single matrix on multiple machines, and access it seamlessly in the way that you propose.

IMHO, allowing for data to be paged to file/swap should be a setting in MATLAB. Unfortunately, that is not how MATLAB's memory model works, and I suspect it is very difficult to implement this on their side. Plus, when this setting is enabled, users will not be protected anymore against making silly mistakes like zeros(1e7) instead of zeros(1e7,1); it will simply seem to hang the system, as MATLAB is busy filling your entire drive with zeros.

Anyway, I think it is possible using MATLAB classes. But I wouldn't recommend it. Note that implementing a proper subsref and subsasgn is *ahum* challenging, plus, it is likely that you'll have to re-implement many algorithms (like mldivide). This will most likely mean you'll lose a great deal of performance; think factors of in the thousands.

Here's an interesting random relevant paper I found while googling around a bit.

Upvotes: 4

Daniel
Daniel

Reputation: 36720

Write a function:

function a=BIG(x,y)
  m = matfile('BIG.mat');
  a = m.BIG(x,y);
end

Every time you write BIG(a,b) the function is called.

Upvotes: 0

Related Questions