Using numpy.array in cython

Question

I want to rewrite a class in cython format and save it as demo.pyx. The input parameter for the class would be either a 2D np.array with an Nx2 shape, e.g. a=np.array([[0.2,-0.8],[3.7,0.02],..,[-0.92,-3.33]]), or a list for instance a=[0.1,2.7].

cimport numpy as np
DTYPE = np.float64
ctypedef np.float64_t DTYPE_t
cdef class halo_positions(object):
    cdef unsigned double *x 
    cdef unsigned double *y 
    def __init__(self, np.ndarray[np.float64_t,ndim=2,mode='c'] positions):
        self.x = &positions[:,0]
        self.y = &positions[:,1]

What I have tried to write causes error message as following:

running build_ext
cythoning demo.pyx to demo.c

Error compiling Cython file:
------------------------------------------------------------
...
cimport numpy as np
DTYPE = np.float64
ctypedef np.float64_t DTYPE_t
cdef class halo_positions(object):
    cdef unsigned double *x 
        ^
------------------------------------------------------------

demo.pyx:5:9: Unrecognised type modifier combination

Error compiling Cython file:
------------------------------------------------------------
...
cimport numpy as np
DTYPE = np.float64
ctypedef np.float64_t DTYPE_t
cdef class halo_positions(object):
    cdef unsigned double *x 
    cdef unsigned double *y 
        ^
------------------------------------------------------------

demo.pyx:6:9: Unrecognised type modifier combination

Error compiling Cython file:
------------------------------------------------------------
...
ctypedef np.float64_t DTYPE_t
cdef class halo_positions(object):
    cdef unsigned double *x 
    cdef unsigned double *y 
    def __init__(self, np.ndarray[np.float64_t,ndim=2,mode='c'] positions):
        self.x = &positions[:,0]
                ^
------------------------------------------------------------

demo.pyx:8:17: Cannot take address of Python variable

Error compiling Cython file:
------------------------------------------------------------
...
cdef class halo_positions(object):
    cdef unsigned double *x 
    cdef unsigned double *y 
    def __init__(self, np.ndarray[np.float64_t,ndim=2,mode='c'] positions):
        self.x = &positions[:,0]
        self.y = &positions[:,1]
                ^
------------------------------------------------------------

demo.pyx:9:17: Cannot take address of Python variable

I know there is a problem with the way I have used pointers, but if I want to keep the type of x and y ambiguous, I need to use this. How could I make my class work?

Answer 1

An answer which I got from cython.group in google works perfectly:

import cython
cimport cython

import numpy as np
cimport numpy as np

DTYPE = np.float64
ctypedef np.float64_t DTYPE_t

cdef class halo_positions(object):

    cdef double [:] _x
    property x:
        def __get__(self):
            return np.array(self._x)
        def __set__(self, np.ndarray[DTYPE_t, ndim=1] x):
            self._x = x

    cdef double [:] _y
    property y:
        def __get__(self):
            return np.array(self._y)
        def __set__(self, np.ndarray[DTYPE_t, ndim=1] y):
            self._y = y

    def __init__(self, np.ndarray[DTYPE_t,ndim=2] positions):
        self._x = positions[:,0]
        self._y = positions[:,1]

    def debug(self):
        print self.x, self.y

Answer 2

When you do positions[:,0] or positions[:,1] you are creating a new, 1D and undeclared buffer in Cython. This is not an element from which you can take the address. The address will correspond to a single value in the array, so you should do something like:

cimport numpy as np
DTYPE = np.float64
ctypedef np.float64_t DTYPE_t
cdef class halo_positions(object):
    cdef double *x 
    cdef double *y 
    def __init__(self, np.ndarray[np.float64_t, ndim=2, mode='c'] positions):
        cdef np.ndarray[np.float64_t, ndim=1] buffx, buffy
        buffx = positions[:,0].copy()
        buffy = positions[:,1].copy()
        self.x = &buffx[0]
        self.y = &buffy[0]

Note that:

• YES, you can take the address of an element in the array and use it as a double * array, such that for b=&positions[0,0]; positions[0,0]==b[0] and positions[0,0+1]==b[1]; when positions is a C-contiguous 2D array. You are not converting an np.ndarray into double *, just accessing its data directly from memory;

• I am using .copy() to guarantee that you have contiguous data in memory. This would be unnecessary if positions was Fortran-contiguous.

Answer 3

There are a couple problems with your code. First, and easiest to fix, is that there is no such thing as an "unsigned double", so you should remove the unsigned to begin with.

Also, &positions[:,0] is not the right syntax to get the address of an array, as the : will return a Python object. You need to do &positions[0,0], which points to the initial element of the array.

It should be pointed out that your __init__ will only work for Numpy arrays, and not lists, as you state you'd like to do. You'll have to convert any lists to arrays beforehand. Also bear in mind that you need to keep an active reference to any arrays you use via their pointers or else you're going to run into messy trouble.

Depending on what you want to do with your code, having pointer variables in your class might not be the safest choice in general, though.

Answer 4

Cython can't convert numpy array to double *, you can use double[:] instead, for example:

cimport numpy as np
cdef class halo_positions(object):
    cdef double *x 
    cdef double *y 
    def __init__(self, double[:] positions):
        self.x = &positions[0]
        self.y = &positions[1]

    def debug(self):
        print self.x[0], self.y[0]

but it's very dangerous:

a = np.array([1.0, 2.0, 3.0, 4.0])
hp = halo_positions(a)
hp.debug()
del a
hp.debug() # x and y is wild pointer now.

maybe you should keep a reference to positions in the halo_positions class.

Answer 5

The & operator takes the address of an object. You want to assign self.x to be the address of positions[0]. So it should be self.x = &position[0]. This reads as set the member of self called x to be the address of the 0th element of positions. What you have tried to do is set the address of x to be something. But you aren't allowed to do that. & is only allowed on the right hand side of equations.