CODEWITHSUNDEEP
matlabconcatenationsparse-matrix
olamundo
olamundo

Reputation: 24981

matlab: out of memory when concatenating sparse matrix with a vector

I create a 3560 x 3560 sparse matrix, A. I then create two 1 X 3560 vectors, S and T. When I run the following code (which concatenates S and T as rows in A and afterwards also as columns in A)

A=[A;S;T];
S=[S 0 0];
T=[T 0 0];
A=[A, S', T'];

The last line produces an out of memory error. I guess I am running out of memory since I have other variables stored, but it seems odd to me that adding two 3560 vectors would be the point in which I am exactly hitting my limit, so I think (or more accurately, wishfully think) that somehow the concatenations aren't done in a smart way... Am I right or is there no hope (except for optimizing other pieces in my code)?

EDIT: At the request of yoda, I am posting the full code. Basically what it does is get a N X N matrix of edge weights between the nodes of a graph, and adds two vectors that will act as a source and sink in a max flow computation.

nbr_sim(nbr_sim<0.8)=0;
A=sparse(size(nbr_sim,1)+2,size(nbr_sim,2)+2);
nelements=size(nbr_sim,1);
A(nbr_sim>0)=nbr_sim(nbr_sim>0);
clear nbr_sim;

S=abs([1 0 0]*n);
T=abs([0 1 0]*n);


A(1:nelements,end-1)=S';
A(1:nelements,end)=T';
A(end-1,1:nelements)=S;
A(end,1:nelements)=T;

Upvotes: 1

Views: 2342

Answers (2)

misssprite
misssprite

Reputation: 626

When dealing with large matrix:

For full matrix, you'd better preallocate memory to avoid memory copy during extending.see why

The sparse case is more complicated, and can be even less efficiency than extending in full matrix, because the elements is stored in a compressed manner. Setting an "inner" entry may cause large memory overwrites(have a look here).

So you'd better edit all the entries in advance and create with sparse() function, rather than call sparse() and then pad the data.

Upvotes: 0

abcd
abcd

Reputation: 42225

EDIT:

As you say you have used considerable resources before this operation, it is entirely likely that you are close to the tipping point, when MATLAB gives you an out of memory error.

Remember that when you grow matrices on the fly either by concatenating or by indexing out of range, MATLAB creates a copy of the matrix in memory. So you're not just using up resources for that extra row, but for a copy of that entire matrix!

Here's an example on my machine where I try to grow a vector that's large enough to tip it over the memory limit.

clear
a=rand(2*10^9+1,1); %#create a large array
whos a
  Name               Size                  Bytes  Class     Attributes

  a         2000000001x1             16000000008  double 

%#Now repeat the same, but by growing the array by one element
clear
a=rand(2*10^9,1);
a=[a;0];

??? Error using ==> vertcat
Out of memory. Type HELP MEMORY for your options.

So you see that although MATLAB can create a matrix with 2*10^9+1 elements in one go, when you try to create an array of the same size by append a single element to a 2*10^9 element vector, it runs out of memory.

If S and T are column vectors as you say, then A=[A;S;T] should give you an error:

??? Error using ==> vertcat CAT arguments dimensions are not consistent.

So you must be doing something else. Concatenating will not change sparseness of the matrix i.e., it won't switch from sparse to full.

A=sprand(3560,3560,0.01); %#test matrices
S=rand(3560,1);
T=rand(3560,1);

B=[A,S,T]; %#join the columns
issparse(B)

ans =

     1

Moreover, a 3560x3560 matrix of doubles is only ~97 MB, which shouldn't give you an "out of memory" error...

Upvotes: 2

Related Questions