CODEWITHSUNDEEP
pythonctypes
urschrei
urschrei

Reputation: 26859

ctypes segmentation fault 11 on OSX 10.10 when passing large arrays

I'm calling into a dylib loaded by ctypes which performs a conversion on input values. As far as I can tell, the dylib is working correctly (I wrote it, and can provide the source), and I'm freeing the memory it "leaks" back to Python. Calling it with lists shorter than 65018 items works as expected, but anything larger than that produces Segmentation fault: 11.

from ctypes import cdll, c_float, Structure, POINTER, c_uint32, c_size_t, c_void_p, cast
from sys import platform
import numpy as np

if platform == "darwin":
    ext = "dylib"
else:
    ext = "so"

lib = cdll.LoadLibrary('target/release/liblonlat_bng.' + ext)

class BNG_FFITuple(Structure):
    _fields_ = [("a", c_uint32),
                ("b", c_uint32)]

class BNG_FFIArray(Structure):
    _fields_ = [("data", c_void_p),
                ("len", c_size_t)]

    # Allow implicit conversions from a sequence of 32-bit unsigned
    # integers.
    @classmethod
    def from_param(cls, seq):
        return seq if isinstance(seq, cls) else cls(seq)

    # Wrap sequence of values. You can specify another type besides a
    # 32-bit unsigned integer.
    def __init__(self, seq, data_type = c_float):
        array_type = data_type * len(seq)
        raw_seq = array_type(*seq)
        self.data = cast(raw_seq, c_void_p)
        self.len = len(seq)

# A conversion function that cleans up the result value to make it
# nicer to consume.
def bng_void_array_to_tuple_list(array, _func, _args):
    res = cast(array.data, POINTER(BNG_FFITuple * array.len))[0]
    drop_bng_array(array)
    return [(i.a, i.b) for i in iter(res)]

convert_bng = lib.convert_to_bng
convert_bng.argtypes = (BNG_FFIArray, BNG_FFIArray)
convert_bng.restype = BNG_FFIArray
convert_bng.errcheck = bng_void_array_to_tuple_list
# cleanup
drop_bng_array = lib.drop_int_array
drop_bng_array.argtypes = (BNG_FFIArray,)
drop_bng_array.restype = None

def convertbng_threaded(lons, lats):
    """ Multi-threaded lon lat to BNG wrapper """
    return convert_bng(lons, lats)

N = 55.811741
E = 1.768960
S = 49.871159
W = -6.379880

num_coords = 65017
lon_ls = list(np.random.uniform(W, E, [num_coords]))
lat_ls = list(np.random.uniform(S, N, [num_coords]))


# segfault with > 65017
convertbng_threaded(lon_ls, lat_ls)

My system: MacBook Air w/4GB of memory, OSX 10.11.2, Python 2.7.10

The only thing I can think of is that because my dylib is allocating all its memory on the stack, I'm overflowing it, but I don't know how to verify this.

Upvotes: 3

Views: 547

Answers (1)

Eryk Sun
Eryk Sun

Reputation: 34260

The res array created in bng_void_array_to_tuple_list is a view on the data -- not a copy. That means you need to evaluate the list comprehension to create the list result before calling drop_bng_array(array).

The following demonstrates that the new array is a view on the original data:

>>> from ctypes import *
>>> arr = (c_char * 3)(*'abc')
>>> ptr = POINTER(c_char)(arr)
>>> new_arr = cast(ptr, POINTER(c_char * 3))[0]
>>> new_arr[:]
'abc'

A change made to the original array is visible in the view:

>>> arr[0] = 'z'
>>> new_arr[:]
'zbc'

Upvotes: 3

Related Questions