CODEWITHSUNDEEP
pythonctypes
urschrei
urschrei

Reputation: 26859

Faster conversion to and from ctypes objects using buffers?

I'm converting lists of floats to ctypes Structure classes with the following fields, before passing them to an FFI function:

FFIArray(Structure):
    _fields_ = [("data", c_void_p),
                ("len", c_size_t)]

    @classmethod
    def from_param(cls, seq):
        return seq if isinstance(seq, cls) else cls(seq)

    def __init__(self, seq, data_type = c_float):
        array_type = data_type * len(seq)
        raw_seq = array_type(*seq)
        self.data = cast(raw_seq, c_void_p)
        self.len = len(seq)

The returned FFIArray objects (always of the same length as the input, and the input values themselves aren't being modified by the FFI function) are being converted into lists of tuples like so:

class FFITuple(Structure):
    _fields_ = [("a", c_uint32),
                ("b", c_uint32)]

def void_array_to_tuple_list(array, _func, _args):
    # this is the errcheck function
    res = cast(array.data, POINTER(FFITuple * array.len))[0]
    res_list = [(i.a, i.b) for i in iter(res)]
    drop_bng_array(array)
    return res_list

This works perfectly, but the conversion steps on __init__ and errcheck are still extremely slow for large lists. Is there any speed advantage to first converting the lists to objects which implement the buffer protocol, and creating the FFIArray objects from those using ctypes from_buffer (or from_buffer_copy, or maybe even memmove?), and then performing the reverse on the returned FFIArray objects?

Upvotes: 3

Views: 1238

Answers (1)

urschrei
urschrei

Reputation: 26859

Converting the list to an array and using from_buffer resulted in a 47.5% decrease in object creation time (test script now runs in 204ms vs 387ms, for a million-item list):

FFIArray(Structure):
    _fields_ = [("data", c_void_p),
                ("len", c_size_t)]

    @classmethod
    def from_param(cls, seq):
        return seq if isinstance(seq, cls) else cls(seq)

    def __init__(self, seq, data_type = c_float):
        array_type = data_type * len(seq)
        try:
            raw_seq = array_type.from_buffer(seq.astype(np.float32))
        except (TypeError, AttributeError):
            try:
                raw_eq = array_type.from_buffer_copy(seq.astype(np.float32))
            except (TypeError, AttributeError):
                raw_seq = array_type.from_buffer(array('f', seq))
        self.data = cast(raw_seq, c_void_p)
        self.len = len(seq)

In addition, modifying the FFI function to return two sequences, and not explicitly combining them into tuples, led to a large speedup:

class ResTuple(Structure):
    """ Container for returned FFI data """
    _fields_ = [("e", FFIArray),
                ("n", FFIArray)]


def void_arrays_to_lists(restuple, _func, _args):
    """ Convert the lon, lat --> BNG FFI result to Python data structures """
    eastings = POINTER(c_uint32 * restuple.e.len).from_buffer_copy(restuple.e)[0]
    northings = POINTER(c_uint32 * restuple.n.len).from_buffer_copy(restuple.n)[0]
    res_list = [list(eastings), list(northings)]
    drop_bng_array(restuple.e, restuple.n)
    return res_list

We then modify our setup slightly with the new classes:

convert_bng = lib.convert_to_bng_threaded
convert_bng.argtypes = (FFIArray, FFIArray)
convert_bng.restype = ResTuple
convert_bng.errcheck = void_arrays_to_lists
# cleanup
drop_bng_array = lib.drop_int_array
drop_bng_array.argtypes = (FFIArray, FFIArray)
drop_bng_array.restype = None

Test Script

import _BNG_FFIArray
import pyproj
import numpy as np

N = 55.811741
E = 1.768960
S = 49.871159
W = -6.379880

bng = pyproj.Proj(init='epsg:27700')
wgs84 = pyproj.Proj(init='epsg:4326')
num_coords = 1000000


def test_speed(lon_ls):
    lon_obj = _BNG_FFIArray(lon_ls)


lons = list(np.random.uniform(W, E, [num_coords]))
res = test_speed(lons)

Upvotes: 2

Related Questions