CODEWITHSUNDEEP
c#matrixout-of-memory
user2080209
user2080209

Reputation: 779

System.OutOfMemoryException. Creating a big matrix

I have a matrix [3,15000]. I need to count covariance matrix for the original matrix and then find its eigenvalues.

This is a part of my code:

double[,] covarianceMatrix = new double[numberOfObjects,numberOfObjects];
for (int n=0; n<numberOfObjects;n++)
    {
    for (int m=0;m<numberOfObjects;m++)
    {
        double sum = 0;
        for (int k=0; k<TimeAndRepeats[i,1]; k++)
        {
            sum += originalMatrix[k,n]*originalMatrix[k,m];
        }
    covarianceMatrix[n,m] = sum/TimeAndRepeats[i,1];
    }
}
alglib.smatrixevd(covarianceMatrix,numberOfObjects,1,true,out eigenValues, out eigenVectors);

NumberOfObjects here is about 15000. When I do my computations for a smaller number of objects everything is Ok, but for all my data I get an exeption. Is it possible to solve this problem?

I am using macOS, x64

My environment is MonoDevelop

Upvotes: 3

Views: 1071

Answers (3)

Justin Pearce
Justin Pearce

Reputation: 5097

double[,] covarianceMatrix = new double[numberOfObjects,numberOfObjects];

You said that your matrix is [3, 15000] and that numberOfObjects is 15000. By this line of code here, you're creating a matrix of [15000, 15000] of doubles

15000 * 15000 = 225000000 doubles at 8 bytes each: 1,800,000,000 bytes or 1.8GB

That's probably why you are running out of memory.

Edit:

According to this question and this question the size of objects in C# cannot be larger that 2GB. The 1.8GB does not count any additional overhead required to reference the items in the array, so that 1.8GB might actually be > 2GB when everything is accounted for (Can't say without the debugging info, someone with more C# experience might have to set me straight on this). You might consider this workaround if you're trying to work with really large array, since statically allocated arrays can get messy.

Upvotes: 4

Tarec
Tarec

Reputation: 3255

Exception name tells you exactly what's the problem. You could use floats instead of doubles to bisect ammount of memory needed. Other option would be to create some class object for a covariance matrix that would save data in a disk file, though you'd need to implement proper mechanisms to operate on it and the performance would be limited aswell.

Upvotes: 0

Swift
Swift

Reputation: 1881

When you create covarianceMatrix, you are creatinf an object of 15000*15000 = 225000000

so you need 1800000000 bytes of memory. it is because of that that you have OutofMemoryException

Upvotes: 0

Related Questions