Unpacking complex nested C structs in Python

Question

This is a follow-up question to my previous question, "Unpacking nested C structs in Python". The structs I am working with have since become more complex and I am once again not sure how to unpack and handle them completely.

On the C side, the relevant part of the header looks like this:

typedef struct {
    uint8_t seq;
    uint8_t type;
    uint16_t flags;
    uint16_t upTimestamp;
}__attribute__ ((packed)) mps_packet_header;

typedef struct {
    mps_packet_header header;
    int16_t x[6];
    int16_t y[6];
    int16_t z[6];
    uint16_t lowTimestmp[6];
}__attribute__ ((packed)) mps_acc_packet_t;
typedef mps_acc_packet_t accpacket_t;

typedef struct {
    int16_t hb1;
    int16_t hb2;
} acam_hb_data_set;

typedef struct __attribute__ ((packed)) {
    mps_packet_header header;
    uint16_t temp;
    uint16_t lowTimestmp[8];
    acam_hb_data_set dms_data[8];
} mps_dms_packet_t;

Two challenges arise from this. First, packets I receive (in binary form) can be either mps_acc_packet_t or mps_dms_packet_t - the only way to tell them apart is to read the type field in the mps_packet_header which both packet types have. This means that I need to unpack the packet before I know its full contents, which I don't know how to do cleanly as (if I'm not mistaken) the two packet types have a different calcsize (54 and 56 respectively). The second challenge is unpacking an mps_dms_packet_t; as you can see from the struct's definition, this packet has an array consisting of 8 instances of acam_hb_data_set, which in turn is a struct consisting of two int16 values. I don't know how to formulate a correct format string for this.

My previous code (before mps_dms_packet_t was introduced) looked like this, as suggested by falsetru in his answer to my previous question:

s = struct.Struct('= B B H H 6h 6h 6h 6H')
fields = s.unpack(packet_data)
seq, _type, flags, upTimestamp = fields[:4]
x = fields[4:10]
y = fields[10:16]
z = fields[16:22]
lowTimestamp = fields[22:]

This worked fine. Now, I need to somehow be able to read the header (for which I need to unpack the struct) and then unpack the struct correctly depending on its type.

How would I go about doing this?

Answer 1

I've written this small package that is based on the bitstruct package. Basically it allows you to do what you can already implement as per the previous answer, but including the possibility to add bit fields in the python side.

For example

typedef struct __attribute__((packed)) {
    uint8_t number;
    char brand[10];
} shoes_t;

typedef struct __attribute__((packed)) {
    char tshirt[10];
    char shorts[10];
    shoes_t shoes;
} clothes_t;

typedef struct __attribute__((packed)) {
    char name[10];
    uint8_t age;
    float weight;
    clothes_t clothes;
} person_t;

is represented as:

shoes_t =  Struct(
    {
        "number": c_unsigned_int(8),
        "brand": c_char(10*8)
    }
)

dresses_t =  Struct(
    {
        "tshirt": c_char(10*8),
        "shorts": c_char(10*8),
        "shoes": shoes_t
    }
)

person_t = Struct(
    {
        "name": c_char(10*8),
        "age": c_unsigned_int(8),
        "weight": c_float(32),
        "dresses": dresses_t
    }
)

The Struct class exposes the pack and unpack methods.

You can find an example of usage here

Answer 2

Try ctypes. It is more readable and easier to pack/unpack data with.

import ctypes

class mps_packet_header(ctypes.Structure):
    _fields_ = [
        ("seq", ctypes.c_uint8),
        ("type", ctypes.c_uint8),
        ("flags", ctypes.c_uint16),
        ("upTimestamp", ctypes.c_uint16)
    ]

class mps_acc_packet_t(ctypes.Structure):
    _fields_ = [
        ("header", mps_packet_header),
        ("x", ctypes.c_int16 * 6),
        ("y", ctypes.c_int16 * 6),
        ("z", ctypes.c_int16 * 6),
        ("lowTimestmp", ctypes.c_uint16 * 6)
    ]

class acam_hb_data_set(ctypes.Structure):
    _fields_ = [
        ("hb1", ctypes.c_int16),
        ("hb2", ctypes.c_int16),
    ]


class mps_dms_packet_t(ctypes.Structure):
    _fields_ = [
        ("header", mps_packet_header),
        ("temp", ctypes.c_uint16),
        ("lowTimestmp", ctypes.c_uint16 * 8),
        ("dms_data", acam_hb_data_set * 8)
    ]