MetaStack
MetaStack

Reputation: 3696

Limitations of python pickle? Can it serialize anything?

I'm writting a little module that must be able to persist anything to disk.

I don't know ahead of time what kind of data will be in a variable so, I need default functionality that can serialize ANYTHING to disk.

I suppose pickle is the best thing to use in this case because it can serialize python objects to disk and everything is an object in python.

def to_binary(self, folder: str):
    import os
    import pickle
    os.makedirs(folder, mode=0o777, exist_ok=True)
    with open(os.path.join(folder, self.name), mode='wb') as f:
        pickle.dump(self.data, f)                      
        
def from_binary(self, folder: str): 
    os.makedirs(folder, mode=0o777, exist_ok=True)
    with open(os.path.join(folder, self.name), mode='rb') as f:
        self.data = pickle.load(f)

Frankly, though, I'm not sure what it's limitations are. seems like I could use this to write a dictionary to disk, a string, anything vanilla python. But what about other objects? like a pandas DataFrame? (of course that has builtin ways to save to disk, but still, would my stuff choke on a DataFrame object just as an example?)

Do you know of anything that would break this? If so, is there an even more general solution for saving data to disk?


Side note, another reason I might want to use this is for 'hashing' purposes. If I get an object that isn't hashable, theoretically I could use pickle to obtain a character stream that is hashable. As long that its serialization process is deterministic that should be sufficient for my purposes. If you see any problem with this ancillary concern, please let me know.

These kinds of things get into much deeper computer science concepts than I truly understand, so I really appreciate your help!

Upvotes: 1

Views: 1233

Answers (1)

Kaldena
Kaldena

Reputation: 36

From my experience the tensorflow library has some issues with having not picklable objects. See for example this github issue.

I actually don't know where exactly this issue comes from, but it definitely is a negative example.

Upvotes: 1

Related Questions