CODEWITHSUNDEEP
python-3.xpandasnumpy
Kevin Vasko
Kevin Vasko

Reputation: 1645

Can't seem to use use pandas to_csv and read_csv to properly read numpy array

The problem seems to stem from when I read in the csv with read_csv having a type issue when I try to perform operations on the nparray. The following is a minimum working example.

x = np.array([0.83151197,0.00444986])
df = pd.DataFrame({'numpy': [x]})
np.array(df['numpy']).mean()

Out[151]: array([ 0.83151197,  0.00444986])

Which is what I would expect. However, if I write the result to a file and then read the data back into a pandas DataFrame the types are broken. 

x = np.array([0.83151197,0.00444986])
df = pd.DataFrame({'numpy': [x]})
df.to_csv('C:/temp/test5.csv')
df5 = pd.read_csv('C:/temp/test5.csv', dtype={'numpy': object})
np.array(df5['numpy']).mean()

TypeError: unsupported operand type(s) for /: 'str' and 'long'

The following is the output of "df5" object

df5 
Out[186]:     
    Unnamed: 0                      numpy 
 0           0    [0.83151197  0.00444986]

The following is the file contents:

,numpy
0,[ 0.83151197  0.00444986]

The only way I have figured out how to get this to work is to read the data and manually convert the type, which seems silly and slow.

[float(num) for num in df5['numpy'][0][1:-1].split()]

Is there anyway to avoid the above?

Upvotes: 0

Views: 887

Answers (1)

MaxU - stand with Ukraine
MaxU - stand with Ukraine

Reputation: 210812

pd.DataFrame({'col_name': data}) expects a 1D array alike objects as data:

In [63]: pd.DataFrame({'numpy': [0.83151197,0.00444986]})
Out[63]:
      numpy
0  0.831512
1  0.004450

In [64]: pd.DataFrame({'numpy': np.array([0.83151197,0.00444986])})
Out[64]:
      numpy
0  0.831512
1  0.004450

you've wrapped numpy array with [] so you passed a list of numpy arrays:

In [65]: pd.DataFrame({'numpy': [np.array([0.83151197,0.00444986])]})
Out[65]:
                      numpy
0  [0.83151197, 0.00444986]

Replace df = pd.DataFrame({'numpy': [x]}) with df = pd.DataFrame({'numpy': x})

Demo:

In [56]: x = np.array([0.83151197,0.00444986])
    ...: df = pd.DataFrame({'numpy': x})
#                                   ^  ^
    ...: df.to_csv('d:/temp/test5.csv', index=False)
    ...:

In [57]: df5 = pd.read_csv('d:/temp/test5.csv')

In [58]: df5
Out[58]:
      numpy
0  0.831512
1  0.004450

In [59]: df5.dtypes
Out[59]:
numpy    float64
dtype: object

Upvotes: 2

Related Questions