Python3 + ctypes callback causes memory leak in simple example

Question

While working on a complex program combining Python 3 code and C++ code using ctypes, I found a memory leak that can easily be reproduced with the stripped down example below.

My C++ code creates a Python object using a callback function. Next, it calls another callback on the Python object, that simply returns its argument. The second callback causes the object's reference count to increase. As a result, the object never gets garbage-collected.

This is the Python code (file bug.py):

import ctypes

CreateObjectCallback = ctypes.CFUNCTYPE( ctypes.py_object )
NoopCallback = ctypes.CFUNCTYPE( ctypes.py_object, ctypes.py_object )

lib = ctypes.cdll.LoadLibrary("./libbug.so")

lib.test.restype = ctypes.py_object
lib.test.argtypes = [ CreateObjectCallback, NoopCallback ]

class Foo:
    def __del__(self):
        print("garbage collect foo");

def create():
    return Foo()

def noop(object):
    return object

lib.test(CreateObjectCallback(create), NoopCallback(noop))

This is the C++ code (file bug.cpp):

#include <python3.6m/Python.h>
#include <iostream>
#include <assert.h>

extern "C" {

  typedef void *(*CreateObjectCallback)();
  typedef void *(*NoopCallback)(void *arg);

  void *test(CreateObjectCallback create, NoopCallback noop)
  {
    void *object = create();
    std::cerr << "ref cnt = " << ((PyObject*)(object))->ob_refcnt << std::endl;
    object = noop(object);
    std::cerr << "ref cnt = " << ((PyObject*)(object))->ob_refcnt << std::endl;
    return object;
  }
}

And here are the commands I use to compile and run:

g++ -O3 -W -Wextra -Wno-return-type -Wall -Werror -fPIC -MMD   -c -o bug.o bug.cpp
g++ -shared -Wl,-soname,libbug.so -o libbug.so bug.o 
python3 bug.py

The output is:

ref cnt = 1
ref cnt = 2

In other words, the call to the noop function incorrectly increases the reference count, and the Foo object is not garbage collected. Without the call to the noop function, the Foo object is garbage collected. The expected output is:

ref cnt = 1
ref cnt = 1
garbage collect foo

Is this a known issue? Does anyone know a work-around or solution? Is this caused by a bug in ctypes?

Answer 1

You're passing around Python objects. One of your objects is passed into your C code, and not passed out, so you are responsible for that reference count. Here's something that works, but I've changed void* to PyObject* since that is what they are:

#include <Python.h>
#include <iostream>
#include <assert.h>

extern "C" {

  typedef PyObject* (*CreateObjectCallback)();
  typedef PyObject* (*NoopCallback)(PyObject* arg);

  __declspec(dllexport) PyObject* test(CreateObjectCallback create, NoopCallback noop)
  {
    // Create the object, with one reference.
    PyObject* object = create();
    std::cerr << "ref cnt = " << object->ob_refcnt << std::endl;

    // Passing object back to Python increments its reference count
    // because the parameter of the function is a new reference.
    // That python function returns an object (the same one), but
    // now you own deleting the reference.
    PyObject* object2 = noop(object);
    Py_DECREF(object2);

    std::cerr << "ref cnt = " << object->ob_refcnt << std::endl;

    // Your return the created object, but now that Python knows
    // it is a Python object instead of void*, it will decref it.
    return object;
  }
}

Here's the Python script I used. You can use the prototypes as decorators for the callback functions. This really matters if the callback needs to live longer than the function it was passed into. When you call the function as you did directly with the callback wrapper, the callback wrapper is destroyed after the function returns because there is no more reference.

I also change to ctypes.PyDLL. This doesn't release the GIL when calling into the C code. Since you're passing around Python objects that seems a good idea.

import ctypes

CreateObjectCallback = ctypes.CFUNCTYPE( ctypes.py_object )
NoopCallback = ctypes.CFUNCTYPE( ctypes.py_object, ctypes.py_object )

lib = ctypes.PyDLL('test')

lib.test.restype = ctypes.py_object
lib.test.argtypes = [ CreateObjectCallback, NoopCallback ]

class Foo:
    def __del__(self):
        print("garbage collect foo");

@CreateObjectCallback
def create():
    return Foo()

@NoopCallback
def noop(object):
    return object

lib.test(create,noop)

Output:

ref cnt = 1
ref cnt = 1
garbage collect foo