Reputation: 64175
How are the different from_buffer() methods implemented for ctypes types?
In the python ctypes module's __init__.py file, the ctypes types are defined as sub-classes of the _SimpleCData. Such as:
from _ctypes import _SimpleCData
...
class c_long(_SimpleCData):
_type_ = "l"
And the _SimpleCData has the class method of from_buffer().
So I think this from_buffer() method should be also inherited by the child-classes.
But it turns out those methods are different because they are at different addresses:
>>> ctypes.c_long.from_buffer
<built-in method from_buffer of _ctypes.PyCSimpleType object at 0x16f8fb0>
>>> ctypes._SimpleCData.from_buffer
<built-in method from_buffer of _ctypes.PyCSimpleType object at 0x7f19f9b09ae0>
How/where are the different from_buffer() methods get attached to each ctypes type class?
ADD 1 - 9:24 AM 12/28/2021
I use c_int.from_address() and c_char.from_address() on the same address, and different values are interpreted. So these methods are indeed different.
>>> cint =ctypes.c_int.from_address(0x2bc9d60)
>>> cchar =ctypes.c_char.from_address(0x2bc9d60)
>>> cchar2 = ctypes.c_char.from_address(0x2bc9d61) # next byte
>>> hex(ctypes.addressof(cint))
'0x2bc9d60'
>>> hex(ctypes.addressof(cchar))
'0x2bc9d60'
>>> hex(ctypes.addressof(cchar2))
'0x2bc9d61'
>>> cint.value # *(int *)(0x2bc9d60) = 0x116 = 278
278
>>> cchar.value # *(char *)(0x2bc9d60) = 0x16
b'\x16'
>>> cchar2.value # *(char *)(0x2bc9d61) = 0x1
b'\x01'
So c_int takes 4 bytes and c_char takes only 1 byte.
How is this polymorphism achieved?
Upvotes: 2
Views: 749
Answers (2)
Reputation: 41116
Listing [Python.Docs]: ctypes - A foreign function library for Python.
Also listing [GitHub]: python/cpython - (v3.9.9) This is Python version 3.9.9 (v3.9.9 is the version I'm going to use as an example). Any (Python) source file that I'm going to mention will be relative to this.
I'm not going to insist on different method addresses as it was already covered: it's not the method address (a method address wouldn't make any sense in Python) but the class object's (that encapsulate the method).
All the simple C type wrappers are defined Lib/ctypes/__init__.py (as already stated), as subclasses of _SimpleCData. The only thing that differs among them is an apparently minor detail: the _type_ class field. Check [Python.Docs]: struct - Interpret strings as packed binary data for possible values.
The "real magic" happens in Modules/_ctypes/_ctypes.c (mostly). Let's work things out from top to bottom (concept-wise, not by position in the file). I'm not going to list line numbers (as one could search for any identifier name that will be mentioned):
CTypes is based on (external) FFI
Simple_Type is the class implementation for _SimpleCData (which is its display name)
PyCSimpleType_Type (_ctypes.PyCSimpleType) is Simple_Type (_SimpleCData)'s meta class (set via Py_SET_TYPE). Example:
[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q070496724]> "e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Scripts\python.exe" Python 3.9.9 (tags/v3.9.9:ccb0e6a, Nov 15 2021, 18:08:50) [MSC v.1929 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import ctypes as ct >>> import _ctypes as _ct >>> >>> SCD = _ct._SimpleCData >>> SCD <class '_ctypes._SimpleCData'> >>> dir(SCD) ['__bool__', '__class__', '__ctypes_from_outparam__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '_b_base_', '_b_needsfree_', '_objects', 'value'] >>> SCD.__class__ <class '_ctypes.PyCSimpleType'>PyCSimpleType_methods are (some of) PyCSimpleType_Type's methods
CDataType_from_address and CDataType_from_buffer are the 2 method implementations
Both of the above methods rely on PyCData_AtAddress (note the type argument). The key line is:
pd = (CDataObject *)((PyTypeObject *)type)->tp_alloc((PyTypeObject *)type, 0);Going back to PyCSimpleType_Type, its instance creator method is PyCSimpleType_new: this is called when creating _SimpleCData class objects, not their instances
Notice the _type_ member (although the final name will differ). Does it ring a bell?
You can also check SIMPLE_TYPE_CHARS for a familiar set of chars
Things can be taken deeper (Modules/_ctypes/stgdict.c), but I think it's enough
So, the c_* classes (simple C type wrappers) don't have separate implementations for those methods, but the implementation lies in their base (meta) class, and they have a different attribute value (which complies with OOP's inheritance concept).
I created a small example.
code00.py:
#!/usr/bin/env python
import ctypes as ct
import sys
def test_from_addr(addr):
print("\nTest from_address(0x{:016X}) method for various simple C types".format(addr))
types = (
ct.c_ulonglong,
ct.c_uint,
ct.c_ushort,
ct.c_ubyte,
)
for typ in types:
print("{:s} representation: {:}".format(typ.__name__, hex(typ.from_address(addr).value)))
def _simple_c_type_factory(spec):
class SimpleCType(ct._SimpleCData):
_type_ = spec
return SimpleCType
def test_simple_types(addr):
print("\nTest simple C type instances creation from address: 0x{:016X}".format(addr))
specs = "QLIHB"
for spec in specs:
typ = _simple_c_type_factory(spec)
print("{:s} representation: {:}".format(spec, hex(typ.from_address(addr).value)))
def main(*argv):
ull0 = ct.c_ulonglong(0x1234567890ABCDEF)
aull0 = ct.addressof(ull0)
test_from_addr(aull0)
test_simple_types(aull0)
if __name__ == "__main__":
print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
64 if sys.maxsize > 0x100000000 else 32, sys.platform))
rc = main(*sys.argv[1:])
print("\nDone.")
sys.exit(rc)
Output:
[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q070496724]> "e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Scripts\python.exe" code00.py Python 3.9.9 (tags/v3.9.9:ccb0e6a, Nov 15 2021, 18:08:50) [MSC v.1929 64 bit (AMD64)] 064bit on win32 Test from_address(0x00000199612CAE08) method for various simple C types c_ulonglong representation: 0x1234567890abcdef c_ulong representation: 0x90abcdef c_ushort representation: 0xcdef c_ubyte representation: 0xef Test simple C type instances creation from address: 0x00000199612CAE08 Q representation: 0x1234567890abcdef L representation: 0x90abcdef I representation: 0x90abcdef H representation: 0xcdef B representation: 0xef Done.
Notes:
- I only used unsigned types because the memory representation is clearer (things are exactly the same for signed ones)
- The results are a bit surprising (the part at the end matches). This is because my CPU (Intel pc064) is little endian. On a big endian system things would be a lot clearer. Check more details on this topic: [SO]: Python struct.pack() behavior (@CristiFati's answer).
Upvotes: 1
Reputation: 44043
This is a bit convoluted so bear with me. In _ctypes.c (version 3.8.5), there is on line 5788:
Py_TYPE(&Simple_Type) = &PyCSimpleType_Type;
Simple_Type.tp_base = &PyCData_Type;
if (PyType_Ready(&Simple_Type) < 0)
return NULL;
Py_INCREF(&Simple_Type);
PyModule_AddObject(m, "_SimpleCData", (PyObject *)&Simple_Type);
This exposes the address of Simple_Type as the name _SimpleCData. Simple_Type (a.k.a. _SimpleCData to the Python world) also inherits from PyCDataType (more on that later). First, the definition of Simple_Type is on line 5040:
static PyTypeObject Simple_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
"_ctypes._SimpleCData",
sizeof(CDataObject), /* tp_basicsize */
0, /* tp_itemsize */
0, /* tp_dealloc */
0, /* tp_vectorcall_offset */
0, /* tp_getattr */
0, /* tp_setattr */
0, /* tp_as_async */
(reprfunc)&Simple_repr, /* tp_repr */
&Simple_as_number, /* tp_as_number */
0, /* tp_as_sequence */
0, /* tp_as_mapping */
0, /* tp_hash */
0, /* tp_call */
0, /* tp_str */
0, /* tp_getattro */
0, /* tp_setattro */
&PyCData_as_buffer, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
"XXX to be provided", /* tp_doc */
(traverseproc)PyCData_traverse, /* tp_traverse */
(inquiry)PyCData_clear, /* tp_clear */
0, /* tp_richcompare */
0, /* tp_weaklistoffset */
0, /* tp_iter */
0, /* tp_iternext */
Simple_methods, /* tp_methods */
0, /* tp_members */
Simple_getsets, /* tp_getset */
0, /* tp_base */
0, /* tp_dict */
0, /* tp_descr_get */
0, /* tp_descr_set */
0, /* tp_dictoffset */
(initproc)Simple_init, /* tp_init */
0, /* tp_alloc */
GenericPyCData_new, /* tp_new */
0, /* tp_free */
};
This is in essence a metaclass used to define new classes. One of the members of this structure (with the comment /* tp_methods */) references the following array defined on line 4996:
static PyMethodDef Simple_methods[] = {
{ "__ctypes_from_outparam__", Simple_from_outparm, METH_NOARGS, },
{ NULL, NULL },
This should be a list of methods that classes created by this metaclass will have. But wait! That is only a single method. Let's look at the class it inherits from, PyCSimpleType_Type on line 2327:
PyTypeObject PyCSimpleType_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
"_ctypes.PyCSimpleType", /* tp_name */
0, /* tp_basicsize */
0, /* tp_itemsize */
0, /* tp_dealloc */
0, /* tp_vectorcall_offset */
0, /* tp_getattr */
0, /* tp_setattr */
0, /* tp_as_async */
0, /* tp_repr */
0, /* tp_as_number */
&CDataType_as_sequence, /* tp_as_sequence */
0, /* tp_as_mapping */
0, /* tp_hash */
0, /* tp_call */
0, /* tp_str */
0, /* tp_getattro */
0, /* tp_setattro */
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
"metatype for the PyCSimpleType Objects", /* tp_doc */
0, /* tp_traverse */
0, /* tp_clear */
0, /* tp_richcompare */
0, /* tp_weaklistoffset */
0, /* tp_iter */
0, /* tp_iternext */
PyCSimpleType_methods, /* tp_methods */
0, /* tp_members */
0, /* tp_getset */
0, /* tp_base */
0, /* tp_dict */
0, /* tp_descr_get */
0, /* tp_descr_set */
0, /* tp_dictoffset */
0, /* tp_init */
0, /* tp_alloc */
PyCSimpleType_new, /* tp_new */
0, /* tp_free */
};
The methods it brings along are PyCSimpleType_methods defined on line 2318:
static PyMethodDef PyCSimpleType_methods[] = {
{ "from_param", PyCSimpleType_from_param, METH_O, from_param_doc },
{ "from_address", CDataType_from_address, METH_O, from_address_doc },
{ "from_buffer", CDataType_from_buffer, METH_VARARGS, from_buffer_doc, },
{ "from_buffer_copy", CDataType_from_buffer_copy, METH_VARARGS, from_buffer_copy_doc, },
{ "in_dll", CDataType_in_dll, METH_VARARGS, in_dll_doc},
{ NULL, NULL },
};
So it appears that the mataclass builds a new class including all of the methods specified by the tp_methods member of the metaclass's struct definition and its base classes. This should result in the from_buffer method having the same address for all objects for all classes created by the metaclass.
Referring back to your post:
>>> ctypes.c_long.from_buffer
<built-in method from_buffer of _ctypes.PyCSimpleType object at 0x16f8fb0>
>>> ctypes._SimpleCData.from_buffer
<built-in method from_buffer of _ctypes.PyCSimpleType object at 0x7f19f9b09ae0>
I now believe you are misinterpreting what is being output. 0x16f8fb0 and 0x7f19f9b09ae0 are not the addresses of the from_buffer methods, which should be the same for both objects, but rather the addresses of the ctypes.PyCSimpleType objects themselves that are implementing these types.
Mark Tolonen had it right, but in case you needed convincing (I did too) ...
Upvotes: 1
Related Questions
- Ctypes - How to parser buffer content?
- Python ctypes behaviour with variable types
- ctypes from buffer - Python
- Python 3 ctypes call to a function that needs an indirect reference to a buffer through another structure
- Questions on how Ctypes (and potentially the python implementation) handle data types?
- Faster conversion to and from ctypes objects using buffers?
- ctypes buffer modification
- Ctypes from_buffer with memoryviews in Python 2.7 and Python 3.4
- python ctypes uint8 buffer usage
- Which methods implement the buffer interface in Python?