CODEWITHSUNDEEP
pythonpandasnumpy
yamini goel
yamini goel

Reputation: 539

Pandas dataframe reading numpy array column as str

I have two Python scripts, one that creates a .csv file and the other one that reads it.

This is how I save the dataframe in the first file:

df['matrix'] = df['matrix'].apply(lambda x: np.array(x))
df.to_csv("Matrices.csv", sep=",", index=False)

The type and shape of df['matrix'].iloc[0] is <class 'numpy.ndarray'> and (24, 60) respectively.

In the second script when I try

print ("type of df['matrix'].iloc[0]", type(df['matrix'].iloc[0]))

The output is type of df['matrix'].iloc[0] <class 'str'>

How can I make sure that df['matrix'] doesn't loose its nature?

Upvotes: 2

Views: 2364

Answers (2)

jezrael
jezrael

Reputation: 863531

If want save and read only numpy array use savetxt and genfromtxt.

If there are multiple columns then use:

Use pickle:

df.to_pickle('file.pkl')
df = pd.read_pickle('file.pkl')

Convert arrays to multiple columns and then write to file:

a = np.array(
[[219,220,221],
 [154,152,14],
 [205,202,192]])

df = pd.DataFrame({'matrix':a.tolist(), 'b':np.arange(len(a))})
print (df)
            matrix  b
0  [219, 220, 221]  0
1   [154, 152, 14]  1
2  [205, 202, 192]  2

df1 = pd.DataFrame(df.pop('matrix').values.tolist(), index=df.index).add_prefix('mat_')
print (df1)
   mat_0  mat_1  mat_2
0    219    220    221
1    154    152     14
2    205    202    192

df = df.join(df1)
print (df)
   b  mat_0  mat_1  mat_2
0  0    219    220    221
1  1    154    152     14
2  2    205    202    192

But if really need to convert values to array need converter with ast.literal_eval:

import ast

df.to_csv('testing.csv', index=False)

df = pd.read_csv('testing.csv', converters={'matrix':lambda x: np.array(ast.literal_eval(x))})
print (type(df.loc[0, 'matrix']))

<class 'numpy.ndarray'>

Upvotes: 2

anky
anky

Reputation: 75130

For saving arrays directly to csv as multiple columns use:

np.savetxt(r'C:\path\file.csv',a,delimiter=',')

If you need to read back as a python object, ast.literal_eval() is your saviour as pointed by @jezrael

Upvotes: 1

Related Questions