CODEWITHSUNDEEP
pythonpandas
David
David

Reputation: 1267

Read a pickle with a different version of pandas

I can't read a pickle file saved with a different version of Python pandas. I know this has been asked here before, but the solutions offered, using pd.read_pickle("my_file.pkl") is not working either. I think (but I am not sure) that these pickle files were created with an newer version of pandas than that of the machine I am working now.

Unfortunately, I am not the administrator and I cannot change the version of pandas. How can I read my files? Are they irrecoverable?

Upvotes: 5

Views: 15120

Answers (3)

kzg
kzg

Reputation: 51

For data frames that contain objects such as lists and arrays in individual cells, instead of converting to a csv file, saving as hdf worked.

Step 1

Load data with the newest version of python and pandas (python 3.8 and pandas 1.4.1 in my case).

import pandas as pd
import pickle
data = pd.read_pickle('path/to/file.pkl')

Step 2

Save the loaded data as a hdf file and set pickle protocol to 4 and key to data frame

pickle.HIGHEST_PROTOCOL = 4
data.to_hdf('output/folder/path/to/file.hdf', 'df')

Step 3

Load the hdf file with the older python (Python 3.7 and pandas 1.3.5 on google colab for me)

import pandas as pd
data = pd.read_hdf(path)

Upvotes: 5

Kenan
Kenan

Reputation: 14104

You will need the same version (or a later one) of pandas as the one used to_pickle. When pandas converts a dataframe to pickle the compress process is specific to that version.

I advise to contact your administrator and have them convert the pickle to csv that way you can open it with any version of pandas.

Unless the dataframe contains objects csv should be fine

Upvotes: 2

foureyes9408
foureyes9408

Reputation: 113

Regarding your question about compatibility in the comments, per the docs related to pickle, "The pickle serialization format is guaranteed to be backwards compatible across Python releases provided a compatible pickle protocol is chosen..." If the files were pickled with a newer protocol than you're reading with, you may be out of luck.

Upvotes: 6

Related Questions