CODEWITHSUNDEEP
pythonpandasdataframebyte
florentine
florentine

Reputation: 657

How to Convert Wrapped Bytes to Actual Bytes in Python Dataframe?

I have a column in my pandas dataframe which stores bytes. I believe the bytes are getting converted to a string when I put it in the dataframe because dataframe doesn't support actual bytes as a dtype. So instead of the column values being b'1a2b', it ends up getting wrapped in a string like this: "b'1a2b'".

I'm passing these values into a method that expects bytes. When I pass it like this ParseFromString("b'1a2b'"), I get the error message:

TypeError: memoryview: a bytes-like object is required, not 'str'

I was confused if encode or decode works in this case or if there is some other way to convert this wrapped bytes into bytes? (I'm using Python 3)

Since these values are in a dataframe, I can use a helper method during the conversion process from string-->bytes--> protocol buffer since the actual dataframe might not be able to store it as bytes. For example, my_dataframe.apply(_helper_method_convert_string_to_bytes_to_protobuf).

Upvotes: 0

Views: 196

Answers (2)

PacketLoss
PacketLoss

Reputation: 5746

You can take a few approaches here, noting that eval() is considered bad practice and it is best to avoid this where possible.

  1. Store your byte representation as a string and encode() on call to function
  2. Extract the byte representation out of your string, then call encode() to function

Whilst if possible, it would be best to just store your bytes as 1a2b when importing the data, if that's not possible you could use regex to extract the contents of the string between b'' and pass the result to encode().

import re

string = "b'1a2b'"

re.search(r"(?<=').*(?=')", string).group().encode()

Output:

#b'1a2b'
type(re.search(r"(?<=').*(?=')", string).group().encode())
#<class 'bytes'>

Upvotes: 0

annicheez
annicheez

Reputation: 187

So the problem seems to be that you are unable to extract the byte object from the string. When you pass the string to the function, which is expecting a byte object like b'1a2b', it throws an error. My suggestion would be to try wrapping your string in an eval function. Like:

a = "b'1a2b'"
b = eval(a)

b is what you want. You haven't shared the code for your function, so I'm unable to do amend the actual code for you.

Upvotes: 1

Related Questions