What is the simple way to extend the use of array to prevent out of memory exception?

Question

float[][, , ,] tempData = new float[30][, , ,];

private void InitTempData()
{
    const int FocusSize = 400;
    try
    {
        for (int i = 0; i < 30; i++)
        {
            tempData[i] = new float[40, FocusSize, FocusSize, 5];
        }
    }
    catch (OutOfMemoryException ex)
    {
        MessageBox.Show(ex.Message);
    }
}

I need to use the tempData with array size of like this:

tempData[30][40, 400, 400, 5]

But what I have experienced so far, it will go OutOfMemory when I define new Array at size more than 100.

What the idea I think of is, I initialize four new Array with size of 100. And use the four new Array starting with different initial counter as per below:

float[][, , ,] tempData0 = new float[30][, , ,];
float[][, , ,] tempData1 = new float[30][, , ,];
float[][, , ,] tempData2 = new float[30][, , ,];
float[][, , ,] tempData3 = new float[30][, , ,];

private void InitTempData()
{
    const int FocusSize = 100;
    try
    {
        for (int i = 0; i < 30; i++)
        {
            tempData0[i] = new float[40, FocusSize, FocusSize, 5];
            tempData1[i] = new float[40, FocusSize, FocusSize, 5];
            tempData2[i] = new float[40, FocusSize, FocusSize, 5];
            tempData3[i] = new float[40, FocusSize, FocusSize, 5];
        }
    }
    catch (OutOfMemoryException ex)
    {
        MessageBox.Show(ex.Message);
    }
}

//Use the tempData0, tempData1, tempData2, and tempData3 with different initial counter
for (int i = 0; i < 30; i++)
{
    for (int x = 0; x < FocusSize; x++)
    {
        for (int z = 0; z < FocusSize; z++)
        {
            //Use tempData0 here
        }
    }
}

for (int i = 0; i < 30; i++)
{
    for (int x = FocusSize; x < FocusSize * 2; x++)
    {
        for (int z = FocusSize; z < FocusSize * 2; z++)
        {
            //Use tempData1 here
        }
    }
}

for (int i = 0; i < 30; i++)
{
    for (int x = FocusSize * 2; x < FocusSize * 3; x++)
    {
        for (int z = FocusSize * 2; z < FocusSize * 3; z++)
        {
            //Use tempData2 here
        }
    }
}

Is my above idea is the correct way of doing it? Or is there any other option to extend the use of array?

Answer 1

IF you are using Windows 32 bit (86x) than you have to check the memory size before adding a new element.

// Here you have to check that the current used memory in your process; this shall not exceed more than 2 GB;
// If you are using  /LARGEADDRESSAWARE than not exceed 4 GB
If (MemoryHelper.CanAllocateObject())
{
 // here add a new elements
}

To get the current process and use:
Process proc = Process.GetCurrentProcess(); proc.PrivateMemorySize64;
http://msdn.microsoft.com/en-us/library/s80a75e5%28VS.80%29.aspx

Answer 2

Does it have to be an array?

I would create a class with a getter and setter. The internal storage can then use smaller arrays.

public class FiveDimensionalData<T> {
    private const int BucketSize = 16384; // keep object out of LOH
    private T[][] _buckets;
    private int _x, _y, _z, _u, _v;

    public FiveDimensionalData(int x, int y, int z, int u, int v) {
        if (x < 0)
            throw new ArgumentOutOfRangeException("x");
        if (y < 0)
            throw new ArgumentOutOfRangeException("y");
        if (z < 0)
            throw new ArgumentOutOfRangeException("z");
        if (u < 0)
            throw new ArgumentOutOfRangeException("u");
        if (v < 0)
            throw new ArgumentOutOfRangeException("v");

        _x = x;
        _y = y;
        _z = z;
        _u = u;
        _v = v;

        long totalSize = ((long)x)*y*z*u*v;

        int totalBuckets = (int)(totalSize / BucketSize) + 1;
        _buckets = new T[totalBuckets][];

        for (int ii = 0; ii < totalBuckets; ++ii)
            _buckets[ii] = new T[BucketSize];
    }

    public T Get(int x, int y, int z, int u, int v) {
        long bucketIndex = Index(x, y, z, u, v); 

        int bucket = (int)(bucketIndex / BucketSize);
        int positionInBucket = (int)(bucketIndex % BucketSize);

        return _buckets[bucket][positionInBucket];
    }

    public void Set(int x, int y, int z, int u, int v, T value) {
        long bucketIndex = Index(x, y, z, u, v); 

        int bucket = (int)(bucketIndex / BucketSize);
        int positionInBucket = (int)(bucketIndex % BucketSize);

        _buckets[bucket][positionInBucket] = value;
    }

    private long Index(int x, int y, int z, int u, int v) {
        if (x < 0 || x > _x)
            throw new ArgumentOutOfRangeException("x");
        if (y < 0 || y > _y)
            throw new ArgumentOutOfRangeException("y");
        if (z < 0 || z > _z)
            throw new ArgumentOutOfRangeException("z");
        if (u < 0 || u > _u)
            throw new ArgumentOutOfRangeException("u");
        if (v < 0 || v > _v)
            throw new ArgumentOutOfRangeException("v");

        long index = (long)x * _y * _z * _u * _v + 
                     (long)y * _z * _u * _v +
                     (long)z * _u * _v +
                     (long)u * _v +
                     (long)v;
        return index;
    }
}            

Answer 3

That is quite a big array:

30 * 40 * 400 * 400 * 5 * (4 bytes) ~= 3.6GB total

Assuming you have that much free memory and are running a 64bit process (required for > 2/3GB), it's possible you are hitting a wall because arrays are laid out as contiguous blocks in memory.

In the example above using [30][40,400,400,5] you're asking the CLR to find 30 128Mb blocks of memory.

See: http://blogs.msdn.com/b/ericlippert/archive/2009/06/08/out-of-memory-does-not-refer-to-physical-memory.aspx

You could try creating lists referencing smaller arrays, or simply declare it as a 'jagged' array to avoid the contiguous memory issue (at the cost of worst performance in dereferencing arrays of arrays).

i.e. float[30][40][400][400][5]