Python ctype - How to pass data between C functions

Question

I have a self-made C library that I want to access using python. The problem is that the code consists essentially of two parts, an initialization to read in data from a number of files and a few calculations that need to be done only once. The other part is called in a loop and uses the data generated before repeatedly. To this function I want to pass parameters from python.

My idea was to write two C wrapper functions, "init" and "loop" - "init" reads the data and returns a void pointer to a structure that "loop" can use together with additional parameters that I can pass on from python. Something like

void *init() {
  struct *mystruct ret = (mystruct *)malloc(sizeof(mystruct));
  /* Fill ret with data */
  return ret;
}

float loop(void *data, float par1, float par2) {
  /* do stuff with data, par1, par2, return result */
}

I tried calling "init" from python as a c_void_p, but since "loop" changes some of the contents of "data" and ctypes' void pointers are immutable, this did not work.

Other solutions to similar problems I saw seem to require knowledge of how much memory "init" would use, and I do not know that.

Is there a way to pass data from one C function to another through python without telling python exactly what or how much it is? Or is there another way to solve my problem?

---

I tried (and failed) to write a minimum crashing example, and after some debugging it turned out there was a bug in my C code. Thanks to everyone who replied! Hoping that this might help other people, here is a sort-of-minimal working version (still without separate 'free' - sorry):

pybug.c:

#include <stdio.h>
#include <stdlib.h>

typedef struct inner_struct_s {
  int length;
  float *array;
} inner_struct_t;

typedef struct mystruct_S {
  int id;
  float start;
  float end;
  inner_struct_t *inner;
} mystruct_t;

void init(void **data) {
  int i;
  mystruct_t *mystruct = (mystruct_t *)malloc(sizeof(mystruct_t));
  inner_struct_t *inner = (inner_struct_t *)malloc(sizeof(inner_struct_t));
  inner->length = 10;
  inner->array = calloc(inner->length, sizeof(float));
  for (i=0; i<inner->length; i++)
    inner->array[i] = 2*i;
  mystruct->id = 0;
  mystruct->start = 0;
  mystruct->end = inner->length;
  mystruct->inner = inner;
  *data = mystruct;
}

float loop(void *data, float par1, float par2, int newsize) {
  mystruct_t *str = data;
  inner_struct_t *inner = str->inner;
  int i;
  inner->length = newsize;
  inner->array  = realloc(inner->array, newsize * sizeof(float));
  for (i=0; i<inner->length; i++)
    inner->array[i] = par1 + i * par2;
  return inner->array[inner->length-1];
}

compile as

cc -c -fPIC pybug.c
cc -shared -o libbug.so pybug.o

Run in python:

from ctypes import *
sl = CDLL('libbug.so')
# What arguments do functions take / return?
sl.init.argtype = c_void_p
sl.loop.restype = c_float
sl.loop.argtypes = [c_void_p, c_float, c_float, c_int]

# Init takes a pointer to a pointer
px = c_void_p()
sl.init(byref(px))

# Call the loop a couple of times
for i in range(10):
    print sl.loop(px, i, 5, 10*i+5)

Answer 1

You should have a corresponding function to free the data buffer when the caller is done. Otherwise I don't see the issue. Just pass the pointer to loop that you get from init.

init.restype = c_void_p
loop.argtypes = [c_void_p, c_float, c_float]
loop.restype = c_float

I'm not sure what you mean by "ctypes' void pointers are immutable", unless you're talking about c_char_p and c_wchar_p. The issue there is if you pass a Python string as an argument it uses Python's private pointer to the string buffer. If a function can change the string, you should first copy it to a c_char or c_wchar array.

Here's a simple example showing the problem of passing a Python string (2.x byte string) as an argument to a function that modifies it. In this case it changes index 0 to '\x00':

>>> import os
>>> from ctypes import *
>>> open('tmp.c', 'w').write("void f(char *s) {s[0] = 0;}")
>>> os.system('gcc -shared -fPIC -o tmp.so tmp.c')
0
>>> tmp = CDLL('./tmp.so')
>>> tmp.f.argtypes = [c_void_p]
>>> tmp.f.restype = None
>>> tmp.f('a')
>>> 'a'
'\x00'
>>> s = 'abc' 
>>> tmp.f(s)
>>> s
'\x00bc'

This is specific to passing Python strings as arguments. It isn't a problem to pass pointers to data structures that are intended to be mutable, either ctypes data objects such as a Structure, or pointers returned by libraries.

Answer 2

Is your C code in a DLL? If so can might consider creating a global pointer in there. init() will do any initialization required and set the pointer equal to newly allocated memory and loop() will operate on that memory. Also don't forget to free it up with a close() function