CODEWITHSUNDEEP
python
user2345778
user2345778

Reputation: 31

Converting data to binary using struct in python

I have the following dict which I want to write to a file in binary: 

data = {(7, 190, 0): {0: 0, 1: 101, 2: 7, 3: 0, 4: 0}, 
        (7, 189, 0): {0: 10, 1: 132, 2: 17, 3: 20, 4: 40}}

I went ahead to use the struct module in this way:

packed=[]
for ssd, add_val in data.iteritems():
    # am trying to using 0xcafe as a marker to tell me where to grab the keys 
    pack_ssd = struct.pack('HBHB', 0xcafe, *ssd) 
    packed.append(pack_ssd)
    for add, val in data[ssd].iteritems():
        pack_add_val = struct.pack('HH', add, val)
        packed.append(pack_add_val)

The output of this is packed = ['\xfe\xca\x07\x00\xbe\x00\x00', '\x00\x00\x00\x00', '\x01\x00e\x00', '\x02\x00\x07\x00', '\x03\x00\x00\x00', '\x04\x00\x00\x00', '\xfe\xca\x07\x00\xbd\x00\x00', '\x00\x00\n\x00', '\x01\x00\x84\x00', '\x02\x00\x11\x00', '\x03\x00\x14\x00', '\x04\x00(\x00']

After which I write this as a binary file :

ifile = open('test.bin', 'wb')
for pack in packed:
    ifile.write(pack)

Here is what the binary file looks like: '\xfe\xca\x07\x00\xbe\x00\x00\x00\x00\x00\x00\x01\x00e\x00\x02\x00\x07\x00\x03\x00\x00\x00\x04\x00\x00\x00\xfe\xca\x07\x00\xbd\x00\x00\x00\x00\n\x00\x01\x00\x84\x00\x02\x00\x11\x00\x03\x00\x14\x00\x04\x00(\x00'

It's all OK until I tried to unpack the data. Now I want to read the contents of the binary file and arrange it back to how my dict looked liked in the first place. This is how I tried to unpack it but I was always getting an error:

unpack=[]
while True:
chunk = ifile.read(log_size)
if len(chunk) == log_size:
    str = struct.unpack('HBHB', chunk)
    unpack.append(str)
    chunk = ifile.read(log1_size)
    str= struct.unpack('HH', chunk)
    unpack.append(str)

Traceback (most recent call last):
File "<interactive input>", line 7, in ?
error: unpack str size does not match format

I realize the method I tried to unpack will always run into problems, but I can't seem to find a good way in unpacking the contents of the binary file. Any help is much appreciated..

Upvotes: 1

Views: 875

Answers (1)

Patashu
Patashu

Reputation: 21793

If you need to write something custom, I would suggest doing the following:

1) 64 bit integer: Number of keys

2) 64 bit integer * 3 * number of keys: Key tuple data

for i in number of keys:

3i) 64 bit integer: Number of keys for dictionary i

4i): 64 bit integer * 2 * number of keys for i: key data, value data, key data, value data...

After that, just make sure you read and write with the same endianness and that specifying an invalid length at any point (too high, too low) doesn't crash your program and you are good.

The idea is that at any state in the unpacker it is either expecting a length or to read data as something, and so it is 100% unambiguous where everything starts and ends as long as you follow the format.

Upvotes: 1

Related Questions