CODEWITHSUNDEEP
pythonpickleopenstackopenstacksdk
Caesar
Caesar

Reputation: 109

Retrieving data with openstacksdk from openstack's object storage

I'm trying to retrieve pickle data I have uploaded to an openstack object storage using openstacksdk's connection.get_object(container,object), I get a response from it, however the file body is a string, I can even save it to file with the outfile option without issues. However I would like to be able to work with it directly without having to resort to save it to file first and then loading it into pickle.

Simply using pickle's load and loads doesn't work as neither takes string objects. Is there another way to retrieve the data so I can work with the pickled data directly or is there some way to parse to string/set a config parameter on get_object()?

Upvotes: 0

Views: 283

Answers (2)

Caesar
Caesar

Reputation: 109

EDIT: I found the solution, for pickled objects or any other files retrieved from openstack with openstacksdk, there are a few ways of dealing with the data without resorting to disk.

First my implemented solution was to use openstack's connection method get_object_raw:

conn = connection(foo,bar, arg**) 
pickle.loads(conn.get_object_raw('containerName', 'ObjectName').content)

.get_object_raw returns a response request object with the attribute content which is the binary file content which is the pickle content one can load with pickle.

You could also create a temporary in-memory file with io.BytesIO, and using it as the outfile argument in get_object from the connection object.

Upvotes: 0

Sean
Sean

Reputation: 681

If you are using Python 3 - pickle expects a bytes-like object. The load method takes a file path, and relies on the file type to handle the providing of bytes back into pickle. When you use the loads method you need to provide it a bytes-like object, not a string, so you will need to convert the string to bytes.

Best way to convert string to bytes in Python 3?

Upvotes: 0

Related Questions