CODEWITHSUNDEEP
pythonccythondynamic-memory-allocationcustom-data-type
Fra
Fra

Reputation: 5188

python iterate over dynamically allocated Cython array

I'm writing a python wrapper to a C class and I'm allocating memory using PyMem_Malloc as explained here

cdef class SomeMemory:

    cdef double* data

    def __cinit__(self, size_t number):
        # allocate some memory (uninitialised, may contain arbitrary data)
        self.data = <my_data_t*> PyMem_Malloc(number * sizeof(my_data_t))
        if not self.data:
            raise MemoryError()

Then I import and use the class in another script using:

import SomeMemory

sm = SomeMemory(10)

I now would like to access the elements of self.data but I encounter 2 problems

  1. if I type self.data and hit enter, the ipython kernel crashes
  2. if I try to loop on self data

like:

for p in self.data:
    print p

I get an error that self.data is not iterable.

How can I access self.data? Do I need to cast the elements to my_data_t first?

Upvotes: 2

Views: 966

Answers (1)

DavidW
DavidW

Reputation: 30901

(Heavily edited in light of the updated question)

So - my understanding is that self.data needs to be declared as public to access it from Python:

cdef class SomeMemory:   
    cdef public double* data
    # also make sure you define __dealloc__ to remove the data

However, that doesn't seem to be enforced here.

However, your real problems are that Python doesn't know what to do with an object of type double *. It's certainly never going to be iterable because the information about when to stop is simply not stored (so it will always go off the end.

There are a range of better alternatives:

Code follows

from cpython cimport array as c_array
from array import array

cdef class SomeMemory:
   cdef c_array.array data

   def __cinit__(self, size_t number):
      self.data = array('d', some initial value goes here)

  • You store your data as a numpy array. This is possibly slightly more complicated, but has much the same advantages. I won't give an example, but it's easier to find.

  • You use Cython's typed memory view: http://docs.cython.org/src/userguide/memoryviews.html. The advantage of this is that it lets you control the memory management yourself (if that's absolutely important to you). However, I'm not 100% sure that you can access them cleanly in Python (too lazy to test!)

  • You wrap the data in your own class implementing __getitem__ and __len__ so it can be used as an iterator. This is likely more trouble that it's worth.

I recommend the first option, unless you have good reason for one of the others.

Upvotes: 2

Related Questions