CODEWITHSUNDEEP
pythonc++optimizationctypes
G.F
G.F

Reputation: 157

Python ctypes 'c_char_p' memory leak

I'm developing a Python library for cryptography. I wanted to optimize my library by writing the main classes in C++ with GMP. I wrote my C++ classes and I wrote the extern methods to use the main arithmetic operations: addition, subtraction, etc... These methods returns the results as char* to avoid cast problems. I built the DLL of my library and I declared the methods in a Python wrapper with ctypes. I noticed that after each arithmetic operation with huge numbers the memory grew exponentially. I was looking for problems in my C++ implementation, but there were no problems thanks to the C++ garbage collector. I was looking for a possible solution, so I discovered that I had to implement a C++ method to free up memory of the string created by the DLL. So I wrote this simple method:

extern "C" {

    __declspec(dllexport) void free_memory(char * n)
    {
        free(n);
    }
    ...
}

I implemented this code in the Python wrapper to free up the memory allocated by the DLL:

import os
import ctypes

DIR_PATH = os.path.dirname(os.path.realpath(__file__))
NUMERIC = ctypes.CDLL(DIR_PATH + "/numeric.dll")
...
NUMERIC.free_memory.argtypes = [ctypes.c_void_p]
NUMERIC.free_memory.restype = None

def void_cast(n):
    a = ctypes.cast(n, ctypes.c_char_p)
    res = ctypes.c_char_p(a.value)
    NUMERIC.free_memory(a)
    return res

So with res = ctypes.c_char_p (a.value) I create a new variable that no longer points to a. This way I correctly delete a using the DLL method, but I still have memory leak problems. It is as if the Python garbage collector does not free up correctly the memory of strings of type c_char_p. In the previous implementation I used only Python and the gmpy2 library, so all the numbers were converted to mpz or mpq. I tested the memory consumption using the memory_profiler package. I created 40 objects of type projective point, defined on an elliptical curve, and I calculated the products i*P, withi from 1 to 40. With gmpy2 about 70MB in total were used. Instead, using ctypes with the classes in C++ the consumption of the memory rose to 1.5GB. It's obvious that there is something wrong, especially when only the base classes that deal with arithmetic operations change. How can I properly free the memory without having memory leak problems? I put an example of extern method for calculating an arithmetic operation, but I have already checked that the problem lies only in correctly freeing the memory via thefree_memory function and reassigning the string so that the garbage collector of Python will free the string when needed.

extern "C" {
    __declspec(dllexport) const char* rat_add(const char * n, const char * m)
    {
        return (RationalNum(n) + RationalNum(m)).getValue();
    }
}

Thanks in advance and have a nice day.

PS: Clearly in C++ I correctly implemented the destructor method to free up the space of the mpz_t and mpq_t objects created.

Upvotes: 3

Views: 1226

Answers (1)

pschill
pschill

Reputation: 5569

The problem is in this line:

res = ctypes.c_char_p(a.value)

This creates a copy of a.value and sets res to a c_char_p that points to the copy. However, Python does not do memory management for ctypes pointers, so the copy will be leaked!

The leak should be fixed if you replace above line with:

res = bytes(memoryview(a.value))

This also creates a copy, but res will be a real Python object.

Upvotes: 2

Related Questions