How to unpack a C-style structure inside another structure?

Question

I am receiving data via socket interface from an application (server) written in C. The data being posted has the following structure. I am receiving data with a client written in Python.

struct hdr
{
   int  Id;
   char PktType;
   int  SeqNo;
   int  Pktlength;
};

struct trl
{
     char Message[16];
     long long info;
};

struct data
{
    char value[10];
    double result;
    long long count;
    short int valueid;
};

 typedef struct
 {
     struct hdr hdr_buf;
     struct data data_buf[100];
     struct trl trl_buf;
 } trx_unit;

How do I unpack the received data to access my inner data buffer?

D.Shawley · Accepted Answer

Using the struct library is the way to go. However, you will have to know a bit more about the C program that is serializing the data. Consider the hdr structure. If the C program is sending it using the naive approach:

struct hdr header;
send(sd, &hdr, sizeof(header), 0);

Then your client cannot safely interpret the bytes that are sent to it because there is an indeterminate amount of padding inserted between the struct members. In particular, I would expect three bytes of padding following the PktType member.

The safest way to approach sending around binary data is to have the server and client serialize the bytes directly to ensure that there is no additional padding and to make the byte ordering of multibyte integers explicit. For example:

/*
 * Send a header over a socket.
 *
 * The header is sent as a stream of packed bytes with
 * integers in "network" byte order.  For example, a
 * header value of:
 *   Id: 0x11223344
 *   PktType: 0xff
 *   SeqNo: 0x55667788
 *   PktLength: 0x99aabbcc
 * 
 * is sent as the following byte stream:
 *   11 22 33 44 ff 55 66 77 88 99 aa bb cc
 */
void
send_header(int sd, struct hdr const* header)
{   /* NO ERROR HANDLING */
    uint32_t num = htonl((uint32_t)header->Id);
    send(sd, &num, sizeof(num), 0);
    send(sd, &header->PktType, sizeof(header->PktType), 0);
    num = htonl((uint32_t)header->SeqNo);
    send(sd, &num, sizeof(num), 0);
    num = htonl((uint32_t)header->PktLength);
    send(sd, &num, sizeof(num), 0);
}

This will ensure that your client can safely decode it using the struct module:

buf = s.recv(13)  # packed data is 13 bytes long
id_, pkt_type, seq_no, pkt_length = struct.unpack('>IBII', buf)

If you cannot modify the C code to fix the serialization indeterminacy, then you will have to read the data from the stream and figure out where the C compiler is inserting padding and manually build struct format strings to match using the padding byte format character to ignore padding values.

I usually write a decoder class in Python that reads a complete value from the socket. In your case it would look something like:

class PacketReader(object):
    def __init__(self, sd):
        self._socket = sd

    def read_packet(self):
        id_, pkt_type, seq_no, pkt_length = self._read_header()
        data_bufs = [self._read_data_buf() for _ in range(0, 100)]
        message, info = self._read_trl()
        return {'id': id_, 'pkt_type': pkt_type, 'seq_no': seq_no,
                'data_bufs': data_bufs, 'message': message,
                'info': info}

    def _read_header(self):
        """
        Read and unpack a ``hdr`` structure.

        :returns: a :class:`tuple` of the header data values
            in order - *Id*, *PktType*, *SeqNo*, and *PktLength*

        The header is assumed to be packed as 13 bytes with
        integers in network byte order.

        """
        buf = self._socket.read(13)
        # >   Multibyte values in network order
        # I   Id as 32-bit unsigned integer value
        # B   PktType as 8-bit unsigned integer value
        # I   SeqNo as 32-bit unsigned integer value
        # I   PktLength as 32-bit unsigned integer value
        return struct.unpack('>IBII', buf)

    def _read_data_buf(self):
        """
        Read and unpack a single ``data`` structure.

        :returns: a :class:`tuple` of data values in order -
            *value*, *result*, *count*, and *value*

        The data structure is assumed to be packed as 28 bytes
        with integers in network byte order and doubles encoded
        as IEEE 754 binary64 in network byte order.

        """
        buf = self._socket.read(28)  # assumes double is binary64
        # >   Multibyte values in network order
        # 10s value bytes
        # d   result encoded as IEEE 754 binary64 value
        # q   count encoded as a 64-bit signed integer
        # H   valueid as a 16-bit unsigned integer value
        return struct.unpack('>10sdqH', buf)

    def _read_trl(self):
        """
        Read and unpack a ``trl`` structure.

        :returns: a :class:`tuple` of trl values in order -
            *Message* as byte string, *info*

        The structure is assumed to be packed as 24 bytes with
        integers in network byte order.

        """
        buf = self.socket.read(24)
        # >   Multibyte values in network order
        # 16s message bytes
        # q   info encoded as a 64-bit signed value
        return struct.unpack('>16sq', buf)

Mind you that this is untested and probably contains syntax errors but that is how I would approach the problem.

How to unpack a C-style structure inside another structure?

Answers (2)

Related Questions