CODEWITHSUNDEEP
python-3.xctypes
SCM
SCM

Reputation: 83

Python Ctypes doesn't compact structure when field and structure fits

not sure if it's normal that Python 3.3.2 alwasy pack structure into a new structure unit. code below demonstrates the problem.

sts structure only occupies 8-bit. pkt only has 24-bit fields before status, so pkt should have size of 32-bit, not 64-bit. the printing of pkt clearly shows that Python packed sts into a new 32-bit integer and left 8-bit unused space with all other pckt field in another 32-bit integer

import ctypes
class sts( ctypes.BigEndianStructure ):
    _fields_ = [( "valid", ctypes.c_uint8, 1 ),
                ( "inout", ctypes.c_uint8, 2 ),
                ( "exception", ctypes.c_uint8, 2 ),
                ( "error", ctypes.c_uint8, 3 ), ]

class pkt( ctypes.BigEndianStructure ):
    _fields_ = [( "uid", ctypes.c_uint32, 8 ),
                ( "sid", ctypes.c_uint32, 16 ),
                ( "sts", sts, ), ]

print("sts {:d}-byte".format(ctypes.sizeof(sts)))
print("pkt {:d}-byte".format(ctypes.sizeof(pkt)))
a=pkt(0xFF,0xDEAD,(0x1,0x3,0x3,0x7))
print("uid {:02X}".format(a.uid))
print("sid {:02X}".format(a.sid))
print("sts {:02X}".format(ctypes.string_at(ctypes.addressof(a.sts))[0]))
for b in ctypes.string_at(ctypes.addressof(a),8):
    print("{:02X}".format(b))

another code to help explain the problem. the output of this code says that Python does pack fields in compact form, but a field with structure always starts on a new unit.

import ctypes

class sts( ctypes.BigEndianStructure ):
    _fields_ = [( "valid", ctypes.c_uint8, 1 ),
                ( "inout", ctypes.c_uint8, 2 ),
                ( "exception", ctypes.c_uint8, 2 ),
                ( "error", ctypes.c_uint8, 3 ), ]

class pkt( ctypes.BigEndianStructure ):
    _fields_ = [( "uid", ctypes.c_uint32, 8 ),
                ( "sid", ctypes.c_uint32, 16 ),
                ( "sts", sts ),
                ( "sts1", sts ),
                ( "gid", ctypes.c_uint16 ), ]

print("sts {:d}-byte".format(ctypes.sizeof(sts)))
print("pkt {:d}-byte".format(ctypes.sizeof(pkt)))
a=pkt(0xFF,0xDEAD,(0x1,0x3,0x3,0x7),(0x1,0x2,0x3,0x7),0xBEEFABCD)
print("uid {:02X}".format(a.uid))
print("sid {:02X}".format(a.sid))
print("sts {:02X}".format(ctypes.string_at(ctypes.addressof(a.sts))[0]))
for b in ctypes.string_at(ctypes.addressof(a),8):
    print("{:02X}".format(b))

Upvotes: 1

Views: 1981

Answers (2)

SCM
SCM

Reputation: 83

use pack and compact fields sovled the problem. Only tried on Windows

import ctypes
class sts( ctypes.BigEndianStructure ):
    _pack_ = 1
    _fields_ = [( "valid", ctypes.c_uint8, 1 ),
                ( "inout", ctypes.c_uint8, 2 ),
                ( "exception", ctypes.c_uint8, 2 ),
                ( "error", ctypes.c_uint8, 3 ), ]

class pkt( ctypes.BigEndianStructure ):
    _pack_ = 1
    _fields_ = [( "uid", ctypes.c_uint8 ),
                ( "sid", ctypes.c_uint16 ),
                ( "sts", sts, ), ]

print("sts {:d}-byte".format(ctypes.sizeof(sts)))
print("pkt {:d}-byte".format(ctypes.sizeof(pkt)))
a=pkt(0xFF,0xDEAD,(0x1,0x3,0x3,0x7))
print("uid {:02X}".format(a.uid))
print("sid {:02X}".format(a.sid))
print("sts {:02X}".format(ctypes.string_at(ctypes.addressof(a.sts))[0]))
for b in ctypes.string_at(ctypes.addressof(a),8):
    print("{:02X}".format(b))

Upvotes: 2

Eryk Sun
Eryk Sun

Reputation: 34270

Microsoft's compiler (or gcc with -mms-bitfields) won't share a storage unit for different integer types. However, with gcc on Linux, pkt does use just 4 bytes. ctypes follows the platform's convention, but you can only use an integer type for a bit field. For example, you can't use ("sts", sts, 8). If your compiler stores pkt in 4 bytes, you'll have to modify the ctypes definition to get the same size. The simplest option is to inline the sts fields in the definition. Using a Union would also work.

Edit:

Due to the way the constructor is written, you have to explicitly set the bits in the tuple, which can only be used with an integer type. It won't look at the contents of a previously defined structure to see whether it can be used to extend an open bitfield. In your example, look at the repr of pkt.uid and pkt.sid. ofs is the offset in bytes and bit offset into the storage unit. On the other hand, for pkt.sts the value of ofs is just 4 bytes -- no bit-field information is set that would be used by the GETBITFIELD and SET macros.

Here are the source links:

Since gcc (except on Windows) automatically tries to pack data to fill the bit-field storage unit (but on a 1-byte boundary, i.e. it won't combine a 28-bit field with a 4-bit field that's defined in another struct), just be careful to double check the layout when using bitfields. To adjust the ctypes definition you can inline fields, explicitly set the number of bits for integer types, or use a union.

FYI, while writing this edit I came across a bug. On non-Windows platforms, PyCField_FromDesc will continue a bit field even if you switch storage size. That's how gcc works on Linux, for example. But I've never actually used it, and this was actually the first time I bothered to look at the GETBITFIELD macro. If bit field switches to a smaller storage size, it uses the getter method for the smaller size combined with the size information (bit offset and number of bits) for the field in the overall context. That can't work in general because the macro ends up bit shifting to the 'left' by a negative number, which is actually a right shift that shifts out all the data, leaving zero. For example:

from ctypes import *

class Test(Structure):
    _fields_ = [
        ('x', c_uint32, 16),
        ('y', c_uint8, 4),
    ]

>>> t = Test.from_buffer_copy(b'\xAA\xAA\xAA\xAA')
>>> bytearray(t)
bytearray(b'\xaa\xaa\xaa\xaa')
>>> hex(t.x)
'0xaaaa'
>>> t.y
0

Upvotes: 1

Related Questions