Dynamic memory allocation for 4 Dimensional c++ array to create HDF5 dataset

Question

I have a program in QT which creates a HDF5 dataset and saves it in disk.

At first I defined this array:

int data[dataNumber][dataFilter][dataWidth][dataHeight];

where dataNumber=400000,dataFilter=1,dataWidth=2,dataHeight=2 and in this loop I set a identical values in this array (of course for simplicity):

for (int i = 0; i < dataNumber; i++)
    for (int j = 0; j < dataFilter; j++)
        for (int k = 0; k < dataWidth; k++)
            for (int l = 0; l < dataHeight; l++)
                data[i][j][k][l]=2;

after that I used these commands to create HDF5 dataset:

// Create a new file using the default property lists.
H5File file(FILE_NAME, H5F_ACC_TRUNC);

datadims[0] = dataNumber;
datadims[1] = dataFilter;
datadims[2] = dataWidth;
datadims[3] = dataHeight;

IntType datatype( PredType::NATIVE_INT );
datatype.setOrder( H5T_ORDER_LE );
DataSpace dataspace(4, datadims);

// Create the dataset.
DataSet dataset = file.createDataSet("data", datatype, dataspace);
dataset.write(data, PredType::NATIVE_INT);

and every thing was ok.

But when I set dataWidth=4,dataHeight=4 ,segmentation fault occured and I think the reserved size of the stack causes this problem and I should use Dynamic memory allocation.

So I changed the data array as follows to overcome with that:

int ****data ;
data = new int***[dataNumber];
for( int i = 0 ; i < dataNumber ; i++ )
{
    data[i] = new int**[dataFilter] ;
    for( int j = 0 ; j < dataFilter ; j++ )
    {
        data[i][j] = new int*[dataWidth] ;
        for( int k = 0 ; k < dataWidth ; k++ )
            data[i][j][k] = new int[ dataHeight ] ;
     }
}

But after using this:

DataSet dataset = file.createDataSet("data", datatype, dataspace);
dataset.write(data, PredType::NATIVE_INT);

the HDF5 dataset had a lot of wrong numbers like -1865382688 and -1720619824 instead of true number=2.

Now I don't know how can I solve this problem. I hope someone can help.

Answer 1

for example we wanna create array (2,2,3,4) like this:

enter code hereint a[2][2][3][3]={{{{1,2,3},{4,5,6},{7,8,9}},{{10,11,12},{13,14,15},{16,17,18}}},{{{1,2,3},{4,5,6},{7,8,9}},{{10,11,12},{13,14,15},{16,17,18}}}};"

so we can allocate memory like

enter code hereint**** data = new int***[2];for(int i = 0;  i < 2; i++){
data[i] = new int**[2];
    for (int j = 0; j < 3; j++){
        data[i][j]=new int*[3];
            for (int k=0;k<3;k++){
                data[i][j][k]= new int[3];
            }
        }
        
    }}

Answer 2

This is to be expected: You are passing an array of pointers (to pointers to pointers) where HDF5 expects a single multidimensional array. (See my answer to this other question for an ASCII graphic on the difference of the memory layout.) You must allocate your memory in a single chunk if you want to be able to pass it to HDF5.

In C++, you need to do it like Marko Popovic suggests:

int* data = new int[dataNumber*dataFilter*dataWidth*dataHeight];

and do the index calculation yourself. I won't repeat that here.

Of course, you can encapsulate the index calculation in some kind of a macro or template, but either way, it's a PITA to work with 4D arrays in C++.

---

However, if you have the option to write this code in C, there is a better alternative: Use a true 4D array.

int (*data)[dataFilter][dataWidth][dataHeight];
data = malloc(dataNumber*sizeof(*data));

Here, data is a pointer to a 3D slap of your 4D array, and you allocate dataNumber of such slaps to get your 4D array. With this, you can simply index the 4D array like you are used to:

data[i][j][k][l];

C will do the index calculation for you.

Assuming that your array sizes are not known at compile time, this approach is not possible in C++, as C++ requires all array lengths to be compile time constants. C does not have this restriction, allowing dynamic array sizes since C99.

Answer 3

I would suggest to allocate one contiguous chunk of memory, like this:

int *data = new int[dataNumber * dataFilter * dataWidth * dataHeight];

This means that you will need to call the new operator once, instead of calling it thousands of times. This will also make the cleanup of all this memory easy with just making one call:

delete []data;

instead of calling delete thousands of times again. After that, you can index into your array with something like this:

data[((i*dataFilter + j)*dataWidth + k)*dataHeight + l];