Presumed incoherence in Python's struct.pack method

Question

I am very confused right now.

If I pack a 7-length binary string, the result is the following:

>>> struct.pack('7s',b'\x1fBLOCK\n')
b'\x1fBLOCK\n'

Moreover, if I pack an unsigned long long, the result is:

>>> struct.pack('1Q',126208)
b'\x00\xed\x01\x00\x00\x00\x00\x00'

But, if I pack both together, the reuslt adds an extra byte:

>>> struct.pack('7s1Q',b'\x1fBLOCK\n',126208)
b'\x1fBLOCK\n\x00\x00\xed\x01\x00\x00\x00\x00\x00'

Anyone knows why this extra byte appears?

b'\x1fBLOCK\n \x00\x00\xed\x01\x00\x00\x00\x00\x00'

This fact is ruining the binary reading of a custom file...

Answer 1

The layout of bytes produced by struct.pack will (by default) match that produced by your platforms C compiler, which may include pad bytes between fields. You can disable this behaviour by adding a = to the start of your format string:

> struct.pack('7s1Q',b'\x1fBLOCK\n',126208) # C-style layout with padding bytes
'\x1fBLOCK\n\x00\x00\xed\x01\x00\x00\x00\x00\x00'
> struct.pack('=7s1Q',b'\x1fBLOCK\n',126208) # No padding
'\x1fBLOCK\n\x00\xed\x01\x00\x00\x00\x00\x00'

Answer 2

It seems that I have used the @ flag, which means that the byte order is the native one, and the final size is thus variable.

The solution lies on using a fixed size flag, such as <, >, ! or =:

>>> struct.pack('<7s1Q',b'\x1fBLOCK\n',126208)
b'\x1fBLOCK\n\x00\xed\x01\x00\x00\x00\x00\x00'

Answer 3

The additional \x00 is the string termination byte - in C a string is ended by \x00.

You concattenate a string to a unsigned long long, so

Note By default, the result of packing a given C struct includes pad bytes in order to maintain proper alignment for the C types involved

applies.

https://docs.python.org/3/library/struct.html#format-characters

b'\x1fBLOCK\n\x00   \x00\xed\x01\x00\x00\x00\x00\x00'  
    1 23456 7  8th   1   2   3   4   5   6    7   8