CODEWITHSUNDEEP
pythoniosavepygamepickle
Matthew Fournier
Matthew Fournier

Reputation: 1155

Python Pickle crashing on load in Windows

I've got a program that uses Pickle to save and load objects from disk. The pickle saving and loading looks like this:

def saveData(self,obj):
    f = open(os.path.join(self.directory,obj.name),'wb')
    pickle.dump(obj, f)

def loadData(self,fname):
    f = open(os.path.join(self.directory,fname),'rb')
    ret = pickle.load(f)
    return ret

And I have a test method that simply makes an object, saves it, then immediately loads it. In Linux, the object works fine, but in Windows, the program crashes when it tries to load the object. I've re-made the pickled file in Windows, so it isn't trying to open the Linux object dump (although if there's a way to do cross-platform pickling, I would like to know about it)

What could be causing this program to crash in Windows but not Linux?

Upvotes: 2

Views: 3175

Answers (3)

stefanbschneider
stefanbschneider

Reputation: 6086

I had a similar issue that my Python interpreter simply crashed when trying to load a pickled data frame with Pandas pd.read_pickle(). I solved it by updating Pandas with pip install -U pandas (now version 1.2.4).

Upvotes: 1

sophros
sophros

Reputation: 16660

The issue here is related to the differences in the way Python handles certain objects (e.g. paths) depending on the OS.

The nature of the underlying issue is described in this answer.

In short: pickled files are not transferable between OSes (most of the time).

Upvotes: 0

schlenk
schlenk

Reputation: 7247

Pickle basically only stores the attributes of an object and the classname to recreate it.

If you happen to have a C/C++ wrapped object, the attributes might contain pointers to memory locations (aka C-Pointers). So if you pickle that object, destory the old object and restore it, the pointers point somewhere invalid. If those are used during unpickling, you get a crash. And the crash will happen on one OS or the other depending on how the memory manager reused the address. Windows seems to be more eager to free things.

So, does this happen for some extension object? It should not happen for a pure python class.

Upvotes: 1

Related Questions