CODEWITHSUNDEEP
pythonc++python-embedding
Daniel Bresnahan
Daniel Bresnahan

Reputation: 23

How to create and use a python object stored as a c++ object?

So as the title sort of suggests, I am working on a c++ project where I need to call a python module, save it as an object, and call one of its methods multiple times. Below you can find the code for the class, which contains the python object. Currently, it is implemented inside of a for loop which calls the method multiple times. Instantiating the class works fine, along with the first call to the class. However, upon completion of the first loop of the for loop the program crashes with an error along the lines of "free(): invalid size" or sometimes "double free or corruption". I tried using valgrind to try and track down the memory leak, but I get a lot of traces to pythonCApi calls that I dont really understand. 

#include <python2.7/Python.h>
#include <iostream>
#include <algorithm>
#include "predictor.hpp"

using namespace std;


predictor::predictor()
{
  Py_Initialize();

  pName = PyString_FromString("predictor");

  pModule = PyImport_Import(pName);

  Py_XDECREF(pName);

  if (pModule == nullptr) {
    PyErr_Print();
    std::cerr << "Fails to import the module predictor, check installation.\n";
  }

  // dict is a borrowed reference.
  dict = PyModule_GetDict(pModule);
  if (dict == nullptr) {
    PyErr_Print();
    std::cerr << "Fails to get the dictionary, check predictor installation.\n";
    Py_XDECREF(pModule);
  }
  Py_XDECREF(pModule);

  // Builds the name of a callable class
  python_class = PyDict_GetItemString(dict, "Predictor");
  if (python_class == nullptr || python_class == NULL) {
    PyErr_Print();
    std::cerr << "Fails to get the Python class, check predictor installation.\n";
    Py_XDECREF(dict);
  }
  Py_XDECREF(dict);

  // Creates an instance of the class
  if (PyCallable_Check(python_class)) {
    object = PyObject_CallObject(python_class, nullptr);
    if (object == NULL)
    {
        cerr << "Fails to create object.";
        Py_XDECREF(python_class);
    }
    Py_XDECREF(python_class);
} else {
    PyErr_Print();
    std::cout << "Cannot instantiate the Python class" << std::endl;
    Py_XDECREF(python_class);
}

pMethod = PyString_FromString("predict_all");

}


predictor::~predictor()
{

  Py_XDECREF(pMethod);

  Py_XDECREF(object);
  Py_Finalize();

}


long predictor::predict(string rule)
{

  PyObject *pRule = PyString_FromString(rule.c_str());
  PyObject *value = PyObject_CallMethodObjArgs(object, pMethod, pRule, NULL);

  long endValue = PyInt_AsLong(value);

  if (endValue == -1)
  {
    if(!PyErr_Occurred())
    {

        PyErr_Print();
        cerr << "";

        Py_XDECREF(value);
        Py_XDECREF(pRule); 

        return NULL;
    }

    //PyErr_Print();

  }

  Py_XDECREF(value);
  Py_XDECREF(pRule); 

  return endValue;}

Upvotes: 1

Views: 425

Answers (1)

horstr
horstr

Reputation: 2817

The most critical part of writing Python C/C++ code is getting the reference counting right. Python differentiates between different kinds of references, namely new, stolen and borrowed references.

With every API function you call, you have to the check the documentation to see what kind of reference it returns, if any.

New references belong to the caller, so it's correct to use Py_XDECREF to free the object by decrementing the reference count. Be sure not to call Py_XDECREF more than once unless you incremented the reference count in between. In your error handling, Py_XDECREF(pModule) happens twice, for example, because you do not return in the error case, you simply continue.

Borrowed references are owned by somebody else and the reference count was not incremented for you. So, calling Py_XDECREF is only valid if you incremented the reference count yourself before doing so.

PyModule_GetDict(pModule) returns a borrowed reference. You do not increment the reference count, yet you decrement it later with Py_XDECREF(dict). The same is true for PyDict_GetItemString(dict, "predictor"), it returns a borrowed reference yet you decrement it with Py_XDECREF(python_class).

My assumption is that in both cases (dict, python_class), the borrowed references are owned by the module pModule you imported with PyImport_Import(pName). So it's highly likely that you must not decrement the pModule reference count for as long as you're working with borrowed references owned by pModule. Release pModule with Py_XDECREF once you're not using those borrowed references anymore. Alternatively, you could increment the reference count of the borrowed references but as long as you keep pModule around, that should not be necessary.

Upvotes: 2

Related Questions