CODEWITHSUNDEEP
pythonpandaspickle
JR1
JR1

Reputation: 57

How can I preserve lists in a Pandas DataFrame when saving and loading from file?

I have a Pandas DataFrame with a column consisting of lists. E.g.

Idx  foo  bar  lists
0    ___  ___  [0, 1, 2]
1    ___  ___  [3, 4, 5]

I am unsure how to save this DataFrame to file and reload while preserving the list objects.

When using pd.to_csv, the list becomes a string representation. The eval() function would work, but the DataFrame has 10k+ rows and takes a long time.

When pickling and unpickling, I get the following error: UnpicklingError: invalid load key, ','. Which I believe is referring to the commas within each list.

How can I save and load this file without losing the lists?

Upvotes: 1

Views: 1217

Answers (1)

Sayandip Dutta
Sayandip Dutta

Reputation: 15872

You can use ast.literal_eval to converters keyword in the pd.read_csv function.

>>> from ast import literal_eval
>>> df = pd.read_csv('some.csv',converters={'lists':literal_eval})
>>> df
Idx  foo  bar  lists
0    ___  ___  [0, 1, 2]
1    ___  ___  [3, 4, 5]

Upvotes: 4

Related Questions