python struct "unpack requires a bytes object of length 8" while converting binay to float

Question

I'm currently trying to transform binaries into floats and viceversa using the struct module. My function works with certain values, like 2.0 or 14.0, but it doesn't with values like 1.0.

I took the code from another question, and changed it from python2 to python3.

import struct

def bin_to_float(b):
    """ convert binary string to float """
    bf = int_to_bytes(int(b, 2), 8)  # 8 bytes needed for IEEE 754 binary64
    bf = bytes(bf, 'UTF-8')
    print(bf)
    return struct.unpack('<d', bf)[0]

def int_to_bytes(n, minlen):  # helper function
    """ int/long to byte string """
    nbits = n.bit_length() + (1 if n < 0 else 0)  # plus one for any sign bit
    nbytes = (nbits+7)//8  # number of whole bytes
    bytes = []
    for i in range(nbytes):
        bytes.append(chr(n & 0xff))
        n >>= 8   
    # zero pad if needed
    if minlen > 0 and len(bytes) < minlen:
        bytes.extend((minlen-len(bytes)) * '0')
    bytes.reverse()  # put high bytes at beginning
    return ''.join(bytes)

# tests

def float_to_bin(f):
    """ convert float to binary string """
    ba = struct.pack('>d', f)
    s = ''.join('{:08b}'.format(b) for b in ba)
    # strip off leading zeros
    for i in range(len(s)):
        if s[i] != '0':
             break
    else:  # all zeros
        s = '0'
        i = 0
    return s[i:]

import math

floats = [2.0, 1.0, -14.0, 12.546, math.pi]


for f in floats:
    binary = float_to_bin(f)
    print ('float_to_bin(%f): %r' % (f, binary))
    float = bin_to_float(binary)
    print ('bin_to_float(%r): %f' % (binary, float))
    print ()

The problem seems to be that when I encode the str in bytes, I get 9 bytes instead of 8, but this happens only sometimes. This is the console readings:

float_to_bin(2.000000): '100000000000000000000000000000000000000000000000000000000000000'
b'@\x00\x00\x00\x00\x00\x00\x00'
bin_to_float('100000000000000000000000000000000000000000000000000000000000000'): 0.000000

float_to_bin(1.000000): '11111111110000000000000000000000000000000000000000000000000000'
b'?\xc3\xb0\x00\x00\x00\x00\x00\x00'
Traceback (most recent call last):
  File "C:/Users/arzuffi pc test/Desktop/prova struct.py", line 47, in <module>
    float = bin_to_float(binary)
  File "C:/Users/arzuffi pc test/Desktop/prova struct.py", line 8, in bin_to_float
    return struct.unpack('<d', bf)[0]
struct.error: unpack requires a bytes object of length 8

I can't seem to figure out why this happens and how to prevent it from happening :/

Does anybody knows why this is?

Thank you very much in advance.

EDIT: J.J. Hakala answered brilliantly, so I'll post the answer up here:

import struct

def bin_to_float(b):
    """ convert binary string to float """
    return struct.unpack('<d', struct.pack('<Q', int(b, 2)))[0]

def float_to_bin(f):
    """ convert float to binary string """
    return '{:b}'.format(struct.unpack('<Q', struct.pack('<d', f))[0])

If someone wants to do the same with 32 bits float, this should be good:

import struct

def bin_to_float(b):
    """ convert binary string to float """
    return struct.unpack('<f', struct.pack('<L', int(b, 2)))[0]
def float_to_bin(f):
    """ convert float to binary string """
    return '{:b}'.format(struct.unpack('<L', struct.pack('<f',f))[0])

Thank you very much everybody!!!

Answer 1

If you are using structure in storing value in the binary file, to avoid

struct.error: unpack requires a bytes object of length ...

you may do this.

with open('file_name.txt', mode='rb') as file:

    v = 0 # Is a variable starts from 0
    size_ = 9 # It is the size of structure. In this case('ff?') 4 + 4 + 1 = 9

    while True:
        file.seek(v * size_)

        try:
            data = unpack('ff?', file.read(size_))
            print(data)

        except:
            print('File ends')
            break

        v += 1

Here size_ is a variable depends on the size you pack. Since you are getting

'struct.error: unpack requires a bytes object of length 8'

size_ = 8.

Answer 2

struct.pack / struct.unpack could be used to implement the functions like this

def bin_to_float(b):
    return struct.unpack('<d', struct.pack('<Q', int(b, 2)))[0]

def float_to_bin(f):
    return '{:b}'.format(struct.unpack('<Q', struct.pack('<d', f))[0])

As for the original question,

bf = bytes(bf, 'UTF-8')

seems to be the culprit since it may change the length of bf.

Answer 3

Not sure why you're going to all this trouble really. There are much easier ways to perform this sort of conversion:

import ctypes

# Create pointer classes once up front for brevity/performance later
# Assumes ctypes.sizeof(ctypes.c_double) is 8 bytes; could add assert for this
PDOUBLE = ctypes.POINTER(ctypes.c_double)
PU64 = ctypes.POINTER(ctypes.c_uint64)

def float_to_bin(f):
    d = ctypes.c_double(f)       # Convert to true C double
    pd = PDOUBLE(d)              # Make pointer to it
    pu64 = ctypes.cast(pd, PU64) # Cast pointer to unsigned int type of same size
    return '{:b}'.format(pu64[0]) # Read value as unsigned int and convert to bin

def bin_to_float(b):
    u64 = ctypes.c_uint64(int(b, 2)) # Convert bin form to unsigned int
    pu64 = PU64(ul)                  # Make pointer to it
    pd = ctypes.cast(pu64, PDOUBLE)  # Cast pointer to double pointer
    return pd[0]                     # Read double value as Python float

It could be even shorter, but I wanted to limit the work on each line for illustration purposes.

The precise cause of your problem is a little unclear, but the constant switching between bytes and str, and the trickiness of padding and unpadding properly, the possible off-by-one errors in stuff like calculating the precise bit and byte lengths, etc. could all be messing this up. The code above does the job much more directly and portably, without needing to fiddle with the nitty gritty.