CODEWITHSUNDEEP
pythonctypes
gwk
gwk

Reputation: 716

instantiate python object within a c function called via ctypes

My embedded Python 3.3 program segfaults when I instantiate python objects from a c function called by ctypes.

After setting up the interpreter, I can successfully instantiate a python Int (as well as a custom c extension type) from c main:

#import <Python/Python.h>  

#define LOGPY(x) \
{ fprintf(stderr, "%s: ", #x); PyObject_Print((PyObject*)(x), stderr, 0); fputc('\n', stderr); }

// c function to be called from python script via ctypes.
void instantiate() {
  PyObject* instance = PyObject_CallObject((PyObject*)&PyLong_Type, NULL);
  LOGPY(instance);
}

int main(int argc, char* argv[]) {
  Py_Initialize();
  instantiate(); // works fine

  // run a script that calls instantiate() via ctypes.
  FILE* scriptFile = fopen("emb.py", "r");
  if (!scriptFile) {
    fprintf(stderr, "ERROR: cannot open script file\n");
    return 1;
  }

  PyRun_SimpleFileEx(scriptFile, scriptPath, 1); // close on completion
  return 0;
}

I then run a python script using PyRun_SimpleFileEx. It appears to run just fine, but when it calls instantiate() via ctypes, the program segfaults inside PyObject_CallObject:

import ctypes as ct
dy = ct.CDLL('./emb')
dy.instantiate() # segfaults

lldb output:

instance: 0
Process 52068 stopped
* thread #1: tid = 0x1c03, 0x000000010000d3f5 Python`PyObject_Call + 69, stop reason = EXC_BAD_ACCESS (code=1, address=0x18)
    frame #0: 0x000000010000d3f5 Python`PyObject_Call + 69
Python`PyObject_Call + 69:
-> 0x10000d3f5:  movl   24(%rax), %edx
   0x10000d3f8:  incl   %edx
   0x10000d3fa:  movl   %edx, 24(%rax)
   0x10000d3fd:  leaq   2069148(%rip), %rax       ; _Py_CheckRecursionLimit
(lldb) bt
* thread #1: tid = 0x1c03, 0x000000010000d3f5 Python`PyObject_Call + 69, stop reason = EXC_BAD_ACCESS (code=1, address=0x18)
    frame #0: 0x000000010000d3f5 Python`PyObject_Call + 69
    frame #1: 0x00000001000d5197 Python`PyEval_CallObjectWithKeywords + 87
    frame #2: 0x0000000201100d8e emb`instantiate + 30 at emb.c:9

Why does the call to instantiate() fail from ctypes only? The function only crashes when it calls into the python lib, so perhaps some interpreter state is getting munged by the ctypes FFI call?

Upvotes: 3

Views: 640

Answers (1)

gwk
gwk

Reputation: 716

Thanks to Armin Rigo for the hint. The problem is that libraries loaded via ctypes.CDLL() create functions that release the GIL when calling into the native code. As far as I can tell this means that in order for the native function to call back into python code, it would need to acquire the GIL first using the python C API.

The easier alternative is to use ctypes.PyDLL(), which does not release the GIL (it also checks the python error flag). The documentation says, "Thus, this is only useful to call Python C API functions directly." My code is more indirect, in that I have python code calling into my own C functions, which then call into the python C API, but the problem is the same.

Upvotes: 3

Related Questions