CODEWITHSUNDEEP
pythonarraysnumpysparse-matrix
Omar Charfeddine
Omar Charfeddine

Reputation: 37

Storing a big sparse matrix in memory to compute eigenvalues

For my case it's 40000*40000 elements and I only need this matrix to count the eigenvalues.

My problem is how to store the matrix in memory, so I can find the eigenvalues. Here are the details about the matrix:

A=np.array([((Nx-2)(Nz-2))[1/4],((Nx-2)(Nz-2))[1/2],((Nx-2)(Nz-2))[2],((Nx-2)(Nz-2))[1/2],((Nx-2)(Nz-2))[1/4]])
diags=np.array([-Nx,-1,0,1,Nx])
M=spdiags(A, diags, (Nx-2)(Nz-2), (Nx-2)(Nz-2)).toarray()
alpha=np.linalg.eigvals(M)

Error

File "C:/Users/mr_lu/Downloads/2d.py", line 106, in <module>
  M=spdiags(A, diags, (Nx-2)*(Nz-2), (Nx-2)*(Nz-2)).toarray()
File "C:\Users\mr_lu\PycharmProjects\2d\venv\lib\site-packages\scipy\sparse\base.py", line 878, in toarray
  return self.tocoo(copy=False).toarray(order=order, out=out)
File "C:\Users\mr_lu\PycharmProjects\2d\venv\lib\site-packages\scipy\sparse\coo.py", line 310, in toarray
  B = self._process_toarray_args(order, out)
File "C:\Users\mr_lu\PycharmProjects\2d\venv\lib\site-packages\scipy\sparse\base.py", line 1184, in _process_toarray_args
  return np.zeros(self.shape, dtype=self.dtype, order=order)
ValueError: array is too big; `arr.size * arr.dtype.itemsize` is larger than the maximum possible size.

Upvotes: 1

Views: 283

Answers (1)

alta
alta

Reputation: 353

One of the most important details of working with sparse matrices is to never use dense matrix intermediates! It seems that the memory problem occurs when you call toarray(). This is because you are converting a scipy.sparse matrix into a dense numpy.array. Instead of using the numpy module for calculating the eigenvalues on the dense matrix, try using the scipy.sparse.linalg function eigs to find the eigenvalues of your sparse matrix.

Upvotes: 1

Related Questions