Referencing a C++ allocated object in pybind11

Question

I'm trying to create a python binding with pybind11 that references a C++ instance whose memory is handled on the C++ side. Here is some example code:

import <pybind11/pybind11>

struct Dog {
    void bark() { printf("Bark!\n"); }
};

int main()
{
  auto dog = new Dog;
  Py_Initialize();
  initexample(); // Initialize the example python module for import 

  // TBD - Add binding  between dog and example.dog .

  PyRun_StringFlags("import example\n"
                    "\n"
                    "example.dog.bark()\n"  // Access the C++ allocated object dog.
                    , Py_file_input, main_dict, main_dict, NULL);
  Py_Finalize();
}

I'm stuck on how to create the link between the python example.dog and the C++ dog variable.

I can't use py:class_<Dog>.def(py::init<>()) as that will allocate a new instance of Dog, which is not what I want.

Answer 1

Yes I know this answer is pretty late but the solutions provided before are either outdated or solve the Problem in a obscure way.

The biggest Problem with the other answers is that they use pybind and the raw Python interface at once. With pybind you can use a simpler and nicer interface to the Interpreter.

Following a implementation which should solve your Problem.

The first thing you will notice is that we use the "embed.h" Header file. This gives us the Functions to create embedded Modules.

Further down we use the PYBIND11_EMBEDDED_MODULE instead of the regular PYBIND11_MODULE or the outdated PYBIND11_PLUGIN. This is a Macro specifically for embedding.

The next interesting part are the types we define for our structure. Besides the DogType we also use the shared_ptr<Dog>. This is crucial for dealing with instances. When the main module goes out of Scope and starts cleaning up it needs to know that the Class/Struct was of type shared_ptr otherwise you will get a seg fault (Raw Pointers are not usable here, I personally think this a good thing).

The last thing to point out is that we actually use the class pybind11::scoped_interpreter for our interpreter and don't use the Raw Python interface.

#include"pybind11\pybind11.h"
#include"pybind11\embed.h"

#include<iostream>

namespace py = pybind11;

struct Dog {
    void bark() { std::cout << "Bark!\n";  }
};

PYBIND11_EMBEDDED_MODULE(DogModule, m) {
    // look at the types! we have a shared_ptr<Dog>!
    py::class_<Dog, std::shared_ptr<Dog>>(m, "DogModule")
        .def("bark", &Dog::bark);
}


int main(int argc, char **argv) 
{
    // Create Python Interpreter
    py::scoped_interpreter guard;

    // Create Dog Instance
    std::shared_ptr<Dog> ptr = std::make_shared<Dog>();

    // Import the DogModule & Assign the instance to a name in python
    py::module main = py::module::import("__main__");
    main.import("DogModule");
    main.attr("dogInstance") = ptr;

    // Call the bark method from python
    py::exec("dogInstance.bark()");


    getchar();
    return 0;
}

Answer 2

Since Pybind11 v2.2.0 another approach is possible, using custom constructor wrapping: the python init method needs not call the c++ constructor anymore. You can make it return the c++ singleton instance directly.

Declaration in your case could look like:

   // Declare the singleton methods
   py::class_<Singleton>(m, "Singleton")

      .def("init", [](){
          return std::unique_ptr<Singleton, py::nodelete>(&Singleton::instance());
      });

In python:

myInstance1 = Singleton()
myInstance2 = Singleton()

myInstance1 and myInstance2 point to the same c++ object.

That's basically the same answer as that other question.

Answer 3

I found an answer to my own question. The trick was a combination of the following two concepts:

• Create an independent function that returns the singleton.
• Create binding to the singleton class without binding a constructor.

The following example illustrates the technique:

#include <Python.h>
#include <pybind11/pybind11.h>

namespace py = pybind11;
using namespace pybind11::literals;

// Singleton to wrap
struct Singleton
{
  Singleton() : x(0) {}

  int exchange(int n)  // set x and return the old value
  {
    std::swap(n, x);
    return n;
  }

  // Singleton reference 
  static Singleton& instance()
  {
    static Singleton just_one;
    return just_one;
  }

  int x;
};

PYBIND11_PLUGIN(example) {
    py::module m("example", "pybind11 example plugin");

    // Use this function to get access to the singleton
    m.def("get_instance",
          &Singleton::instance,
          py::return_value_policy::reference,
          "Get reference to the singleton");

    // Declare the singleton methods
    py::class_<Singleton>(m, "Singleton")
      // No init!
      .def("exchange",
           &Singleton::exchange,
           "n"_a,
           "Exchange and return the current value"
           )
      ;

    return m.ptr();
}

int main(int argc, char **argv)
{
  Py_Initialize();

  PyObject* main_module = PyImport_AddModule("__main__");
  PyObject* main_dict = PyModule_GetDict(main_module);

  initexample();

  // Call singleton from c++
  Singleton::instance().exchange(999);

  // Populate the example class with two static pointers to our instance.
  if (PyRun_StringFlags("import example\n"
                        "\n"
                        "example.s1 = example.get_instance()\n"
                        "example.s2 = example.get_instance()\n",
                        Py_file_input, main_dict, main_dict, NULL) == nullptr)
      PyErr_Print();

  // Test referencing the singleton references
  if (PyRun_StringFlags("from example import *\n"
                        "\n"
                        "for i in range(3):\n"
                        "  print s1.exchange(i*2+1)\n"
                        "  print s2.exchange(i*2+2)\n"
                        "print dir(s1)\n"
                        "print help(s1.exchange)\n"
                        ,
                        Py_file_input, main_dict, main_dict, NULL) == nullptr)
      PyErr_Print();

  Py_Finalize();

  exit(0);
}