Ctypes: allocate double** , pass it to C, then use it in Python

Question

EDIT 3

I have some C++ code (externed as C) which I access from python. I want to allocate a double** in python, pass it to the C/C++ code to copy the content of a class internal data, and then use it in python similarly to how I would use a list of lists.

Unfortunately I can not manage to specify to python the size of the most inner array, so it reads invalid memory when iterating over it and the program segfaults.

I can not change the structure of the internal data in C++, and I'd like to have python do the bound checking for me (like if I was using a c_double_Array_N_Array_M instead of an array of pointers).

test.cpp (compile with g++ -Wall -fPIC --shared -o test.so test.cpp )

#include <stdlib.h>
#include <string.h>

class Dummy
{
    double** ptr;
    int e;
    int i;
};

extern "C" {
    void * get_dummy(int N, int M) {
        Dummy * d = new Dummy();
        d->ptr = new double*[N];
        d->e = N;
        d->i = M;
        for(int i=0; i<N; ++i)
        {
            d->ptr[i]=new double[M];
            for(int j=0; j <M; ++j)
            {
                d->ptr[i][j] = i*N + j;
            }
        }
        return d;
    }

    void copy(void * inst, double ** dest) {
        Dummy * d = static_cast<Dummy*>(inst);
        for(int i=0; i < d->e; ++i)
        {
            memcpy(dest[i], d->ptr[i], sizeof(double) * d->i);
        }
    }

    void cleanup(void * inst) {
        if (inst != NULL) {
            Dummy * d = static_cast<Dummy*>(inst);
            for(int i=0; i < d->e; ++i)
            {
                delete[] d->ptr[i];
            }
            delete[] d->ptr;
            delete d;
        }
    }


}

Python (this segfaults. Put it in the same dir in which the test.so is)

import os
from contextlib import contextmanager
import ctypes as ct

DOUBLE_P = ct.POINTER(ct.c_double)
library_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'test.so')
lib = ct.cdll.LoadLibrary(library_path)
lib.get_dummy.restype = ct.c_void_p
N=15
M=10

@contextmanager
def work_with_dummy(N, M):
    dummy = None
    try:
        dummy = lib.get_dummy(N, M)
        yield dummy
    finally:
        lib.cleanup(dummy)

with work_with_dummy(N,M) as dummy:
    internal = (ct.c_double * M)
    # Dest is allocated in python, it will live out of the with context and will be deallocated by python
    dest = (DOUBLE_P * N)()
    for i in range(N):
        dest[i] = internal()
    lib.copy(dummy, dest)

#dummy is not available anymore here. All the C resources has been cleaned up
for i in dest:
    for n in i:
        print(n) #it segfaults reading more than the length of the array

What can I change in my python code so that I can treat the array as a list? (I need only to read from it)

Answer 1

3 ways to pass a int** array from Python to C and back

So that Python knows the size of the array when iterating

---

The data

This solutions work for either 2d array or array of pointers to arrays with slight modifications, without the use of libraries like numpy.

I will use int as a type instead of double and we will copy source, which is defined as

N = 10;
M = 15;
int ** source = (int **) malloc(sizeof(int*) * N);
for(int i=0; i<N; ++i)
{
    source[i] = (int *) malloc(sizeof(int) * M);
    for(int j=0; j<M; ++j)
    {
        source[i][j] = i*N + j;
    }
}

---

1) Assigning the array pointers

Python allocation

dest = ((ctypes.c_int * M) * N) ()
int_P = ctypes.POINTER(ctypes.c_int)
temp = (int_P * N) ()
for i in range(N):
    temp[i] = dest[i]
lib.copy(temp)
del temp
# temp gets collected by GC, but the data was stored into the memory allocated by dest

# You can now access dest as if it was a list of lists
for row in dest:
    for item in row:
        print(item)

C copy function

void copy(int** dest)
{
    for(int i=0; i<N; ++i)
    {
        memcpy(dest[i], source[i], sizeof(int) * M);
    }
}

Explanation

We first allocate a 2D array. A 2D array[N][M] is allocated as a 1D array[N*M], with 2d_array[n][m] == 1d_array[n*M + m]. Since our code is expecting a int**, but our 2D array in allocated as a int *, we create a temporary array to provide the expected structure.

We allocate temp[N][M], and than we assign the address of the memory we allocated previously temp[n] = 2d_array[n] = &1d_array[n*M] (the second equal is there to show what is happening with the real memory we allocated).

If you change the copying code so that it copies more than M, let's say M+1, you will see that it will not segfault, but it will override the memory of the next row because they are contiguous (if you change the copying code, remember to add increase by 1 the size of dest allocated in python, otherwise it will segfault when you write after the last item of the last row)

---

2) Slicing the pointers

Python allocation

int_P = ctypes.POINTER(ctypes.c_int)
inner_array = (ctypes.c_int * M)
dest = (int_P * N) ()
for i in range(N):
    dest[i] = inner_array()
lib.copy(dest)

for row in dest:
    # Python knows the length of dest, so everything works fine here
    for item in row:
        # Python doesn't know that row is an array, so it will continue to read memory without ever stopping (actually, a segfault will stop it)
        print(item)

dest = [internal[:M] for internal in dest]

for row in dest:
    for item in row:
        # No more segfaulting, as now python know that internal is M item long
        print(item)

C copy function

Same as for solution 1

Explanation

This time we are allocating an actual array of pointers of array, like source was allocated.

Since the outermost array ( dest ) is an array of pointers, python doesn't know the length of the array pointed to (it doesn't even know that is an array, it could be a pointer to a single int as well).

If you iterate over that pointer, python will not bound check and it will start reading all your memory, resulting in a segfault.

So, we slice the pointer taking the first M elements (which actually are all the elements in the array). Now python knows that it should only iterate over the first M elements, and it won't segfault any more.

I believe that python copies the content pointed to a new list using this method ( see sources )

---

2.1) Slicing the pointers, continued

Eryksun jumped in in the comments and proposed a solution which avoids the copying of all the elements in new lists.

Python allocation

int_P = ctypes.POINTER(ctypes.c_int)
inner_array = (ctypes.c_int * M)
inner_array_P = ctypes.POINTER(inner_array)
dest = (int_P * N) ()
for i in range(N):
    dest[i] = inner_array()
lib.copy(dest)

dest_arrays = [inner_array_p.from_buffer(x)[0] for x in dest]

for row in dest_arrays:
    for item in row:
        print(item)

C copying code

Same as for solution 1

---

3) Contiguous memory

This method is an option only if you can change the copying code on the C side. source will not need to be changed.

Python allocation

dest = ((ctypes.c_int * M) * N) ()
lib.copy(dest)

for row in dest:
    for item in row:
        print(item)

C copy function

void copy(int * dest) {
    for(int i=0; i < N; ++i)
    {
        memcpy(&dest[i * M], source[i], sizeof(int) * M);
    }
}

Explanation

This time, like in case 1) we are allocating a contiguous 2D array. But since we can change the C code, we don't need to create a different array and copy the pointers since we will be giving the expected type to C.

In the copy function, we pass the address of the first item of every row, and we copy M elements in that row, then we go to the next row.

The copy pattern is exactly as in case 1), but this time instead of writing the interface in python so that the C code receives the data how it expects it, we changed the C code to expect the data in that precise format.

If you keep this C code, you'll be able to use numpy arrays as well, as they are 2D row major arrays.

---

All of this answer is possible thanks the great (and concise) comments of @eryksun below the original question.