CODEWITHSUNDEEP
pythonpandas
user1642513
user1642513

Reputation:

Pandas to_csv drops values

I ran into an issue where pandas.to_csv drops values on columns of datetime64 type. 

In [24]: df
Out[24]: 
<class 'pandas.core.frame.DataFrame'>
Int64Index: 28982 entries, 0 to 28981
Data columns (total 4 columns):
value    28982  non-null values
date1    28982  non-null values
date2    22772  non-null values
date3    28982  non-null values
dtypes: datetime64[ns](3), float64(1)

In [25]: df.tail()
Out[25]: 
       value               date1               date2               date3
28977  25.44 2002-08-21 00:00:00 2013-05-03 00:00:00 2007-09-01 00:00:00
28978  25.86 2002-08-21 00:00:00 2013-05-03 00:00:00 2007-09-01 00:00:00
28979  26.08 2002-08-21 00:00:00 2013-05-03 00:00:00 2007-09-01 00:00:00
28980  25.84 2002-08-21 00:00:00 2013-05-03 00:00:00 2007-09-01 00:00:00
28981  25.35 2002-08-21 00:00:00 2013-05-03 00:00:00 2007-09-01 00:00:00

In [26]: df.to_csv('test.csv', index = False)

In [27]: df2 = pd.read_csv('test.csv', header = 0)

In [28]: df2
Out[28]: 
<class 'pandas.core.frame.DataFrame'>
Int64Index: 28982 entries, 0 to 28981
Data columns (total 4 columns):
value    28982  non-null values
date1    28982  non-null values
date2    21070  non-null values
date3    17036  non-null values
dtypes: float64(1), object(3)

In [29]: df2.tail()
Out[29]: 
       value                date1 date2 date3
28977  25.44  2002-08-21 00:00:00   NaN   NaN
28978  25.86  2002-08-21 00:00:00   NaN   NaN
28979  26.08  2002-08-21 00:00:00   NaN   NaN
28980  25.84  2002-08-21 00:00:00   NaN   NaN
28981  25.35  2002-08-21 00:00:00   NaN   NaN

As shown, I wrote df to file and immediately read it back into df2, the columns date2 and date3 in the csv file have a lot of missing values towards the bottom. Is this a bug? By the way I am using Pandas 0.11.

Upvotes: 2

Views: 2606

Answers (1)

Jeff
Jeff

Reputation: 128948

this is a known issue: https://github.com/pydata/pandas/issues/3062

workaround is basically this:

for c in datetime_columns_that_have_NaT:

     df[c] = df[c].astype('object')

df.to_csv()

when you read it back if you specifiy parse_dates=[that_column_num]

it will work

alternatively, you can write like you are and then read like this:

dfc = pd.read_csv('test.csv',index_col=0).convert_objects(convert_dates='coerce')

will force date conversion

Upvotes: 1

Related Questions