CODEWITHSUNDEEP
pythoncctypes
ryan
ryan

Reputation: 153

Using ctypes to pass around struct pointers

So what I have done so far is build a small ctypes and python code which does the following:

So far I have the first 3 stages working but I am having problems with the last two parts. Here is the C code:

#include <stdio.h>
#include <stdlib.h>
//#include "runSolver.h"

#define SMB_MAX_DATA_SIZE 16

typedef struct testStruct {
  double *x[11];
  double *u[10];
} Test;

typedef struct returnStruct_t {
  Test* vars;
} ReturnStruct;

void initalize_returnStruct(void** returnStruct){
  ReturnStruct* new_returnStruct = (ReturnStruct *)malloc(sizeof(ReturnStruct));
  Test* varsStruct = (Test*)malloc(sizeof(Test)*3);

  int dataSize = 5;
  int i;

  for(i = 0; i < 3; i++){
    int x;
    for(x = 0; x < 11; x++)
      varsStruct[i].x[x] = (double *)malloc(sizeof(double)*5);    
    for(x = 0; x < 10; x++)
      varsStruct[i].u[x] = (double *)malloc(sizeof(double)*5);    
  }
  new_returnStruct->vars = varsStruct;
  *returnStruct = new_returnStruct;
}

void free_returnStruct(void* data){
  ReturnStruct* returnStruct = data;
  int i;
  for(i = 0; i < 3; i++){
    int x;
    for(x = 1; x < 11; x++) 
      free(returnStruct->vars[i].x[x]);
    for(x = 0; x < 10; x++)
      free(returnStruct->vars[i].u[x]);
  }
  free(returnStruct->vars);
  free(returnStruct);
}

void parallelSolver(void* data){

  ReturnStruct* VarsArray = data;

  fprintf(stderr, " This is the value: %f \n", VarsArray->vars[0].x[0][0]);  
  fprintf(stderr, " This is the value: %f \n", VarsArray->vars[0].x[10][4]);
  fprintf(stderr, " This is the value: %f \n", VarsArray->vars[0].u[0][0]);
  fprintf(stderr, " This is the value: %f \n", VarsArray->vars[0].u[9][2]);

  VarsArray->vars[0].x[0][0] += 20.0;
  VarsArray->vars[0].x[10][4] += 203.0;
  VarsArray->vars[0].u[0][0] += 202.0;
  VarsArray->vars[0].u[9][2] += 202.0;                         
}

And here is the python code:

#!/usr/bin/python

import sys
import ctypes as ct

numOpt = 3

class vars_t(ct.Structure):
    _fields_ = [("u", ct.POINTER(ct.c_double*10)),
                    ("x", ct.POINTER(ct.c_double*11))]

class returnStruct_t(ct.Structure):
    _fields_ = [("vars", vars_t*numOpt)]

runSolver = ct.CDLL('./runSolverParallel.so')

returnStructPointer = ct.POINTER(returnStruct_t)
runSolver.parallelSolver.argtypes = [ct.c_void_p()]

varsd = ct.c_void_p()
runSolver.initalize_returnStruct(ct.byref(varsd)) 

runSolver.parallelSolver(varsd)
runSolver.parallelSolver(varsd)
runSolver.parallelSolver(varsd)
runSolver.parallelSolver(varsd)

varsdb = ct.cast(varsd, returnStruct_t)

print(varsdb.contents.vars[0].x[0][0])

runSolver.free_returnStruct(varsd)

The code runs fine till I get to these three lines:

varsdb = ct.cast(varsd, returnStruct_t)

print(varsdb.contents.vars[0].x[0][0])

runSolver.free_returnStruct(varsd)

All of which create seg faults. Any advice on how to get this working properly would appreciated!

The errors look like this:

Starting program: /usr/bin/python UserDefinedCode.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
 This is the value: 0.000000 
 This is the value: 0.000000 
 This is the value: 0.000000 
 This is the value: 0.000000 
 This is the value: 20.000000 
 This is the value: 203.000000 
 This is the value: 202.000000 
 This is the value: 202.000000 
 This is the value: 40.000000 
 This is the value: 406.000000 
 This is the value: 404.000000 
 This is the value: 404.000000 
 This is the value: 60.000000 
 This is the value: 609.000000 
 This is the value: 606.000000 
 This is the value: 606.000000 

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff33795d4 in ?? () from /usr/lib/python2.7/lib-dynload/_ctypes.so
(gdb) where
#0  0x00007ffff33795d4 in ?? () from /usr/lib/python2.7/lib-dynload/_ctypes.so
#1  0x00007ffff3386ea4 in ffi_call_unix64 () from /usr/lib/python2.7/lib-dynload/_ctypes.so
#2  0x00007ffff33868c5 in ffi_call () from /usr/lib/python2.7/lib-dynload/_ctypes.so
#3  0x00007ffff33772c2 in _ctypes_callproc () from /usr/lib/python2.7/lib-dynload/_ctypes.so
#4  0x00007ffff3377aa2 in ?? () from /usr/lib/python2.7/lib-dynload/_ctypes.so
#5  0x00000000004d91b6 in PyObject_Call ()
#6  0x000000000054c0da in PyEval_EvalFrameEx ()
#7  0x000000000054c272 in PyEval_EvalFrameEx ()
#8  0x0000000000575d92 in PyEval_EvalCodeEx ()
#9  0x00000000004c1352 in PyRun_SimpleFileExFlags ()
#10 0x00000000004c754f in Py_Main ()
#11 0x00007ffff68cb76d in __libc_start_main (main=0x41ba10 <main>, argc=2, ubp_av=0x7fffffffe1d8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe1c8)
    at libc-start.c:226
#12 0x000000000041ba41 in _start ()

Upvotes: 1

Views: 1200

Answers (1)

abarnert
abarnert

Reputation: 365587

You have at least four problems here (actually, five, but one isn't relevant).

The line that was (usually) causing your segfault is this:

varsdb = ct.cast(varsd, returnStruct_t)

This is because you're trying to cast a void * to a returnStruct_t, rather than to a returnStruct_t *. Since a returnStruct_t is much larger than a pointer, there's a good chance this will run off the end of an allocated page. Even if it doesn't segfault, it's giving you garbage values. It's equivalent to this C code:

returnStruct_t varsdb = *(returnStruct_t *)(&varsd);

What you wanted was the equivalent of:

returnStruct_t *varsdb = (returnStruct_t *)(varsd);

In other words:

varsdb = ct.cast(varsd, returnStructPointer)

After fixing that, I often, but not always, still get a segfault later, on trying to access varsdb.contents.vars[0].x[0][0] (varsdb.contents.vars[0].x[0] itself is always safe).

The next problem is that you haven't defined your struct properly. Here's the C:

typedef struct testStruct {
  double *x[11];
  double *u[10];
} Test;

And here's the Python:

class vars_t(ct.Structure):
    _fields_ = [("u", ct.POINTER(ct.c_double*10)),
                    ("x", ct.POINTER(ct.c_double*11))]

You've mixed up u and x. So, what you're calling x, and treating as an array of 11 doubles, is actually u, an array of 10 doubles. So every time you touch x[10], you're going past the end of the array.

After fixing that one too, I get garbage values printed out. With the clang build, it's always close to 6.92987533417e-310.

I think this one is purely in the C code. I often get garbage numbers printed out from those x[10][4] and u[9][2] lines in C. Again, with the same build, I get about an equal mix of reasonable values, numbers like 26815615859885194199148049996411692254958731641184786755447122887443528060147093953603748596333806855380063716372972101707507765623893139892867298012168192.000000 for both, and reasonable values for the former but nan for the latter.

And when I run a simple C driver under valgrind, every fourth fprintf I get this:

==85323== Use of uninitialised value of size 8

So you've presumably got an off-by-one error in your allocation or initialization code in C, and you sometimes but not always get away with it.

Also, these aren't the same type:

typedef struct returnStruct_t {
  Test* vars;
} ReturnStruct;

class returnStruct_t(ct.Structure):
    _fields_ = [("vars", vars_t*numOpt)]

The former is a single pointer long, and that pointer points at an array of Test objects. The latter is 3 Test objects. So, once again, you're trying to read a pointer to something as a value of that type, and here you're going way past the end of the allocated value.

After fixing that, I no longer get any crashes, and I get reasonable final values like 80.0 even when I get the garbage printouts along the way. But of course I still get those garbage printouts along the way, and valgrind is still complaining, so obviously this still isn't the last problem.

You've also got an obvious leak in your code—which isn't directly relevant to the problem, but it's a good sign that you've probably got similar errors elsewhere. You allocate the x arrays like this:

for(x = 0; x < 11; x++)
  varsStruct[i].x[x] = (double *)malloc(sizeof(double)*5);

… then free them like this:

for(x = 1; x < 11; x++) 
  free(returnStruct->vars[i].x[x]);

So x[0] never gets freed.

Upvotes: 2

Related Questions