CODEWITHSUNDEEP
pythonpandaspostgresql
stargazer
stargazer

Reputation: 33

same pandas dataframes are not equal after storing it to database

I have a pandas dataframe: (edit: actual df)

date
2019-11-01 07:30:00+00:00    1148.144708
2019-11-01 07:45:00+00:00    1148.015876
2019-11-01 08:00:00+00:00    1147.911068
2019-11-01 08:15:00+00:00    1147.925240
2019-11-01 08:30:00+00:00    1148.007117
2019-11-01 08:45:00+00:00    1148.122853
2019-11-01 09:00:00+00:00    1148.251176
2019-11-01 09:15:00+00:00    1148.362736
2019-11-01 09:30:00+00:00    1148.423236
2019-11-01 09:45:00+00:00    1148.528360
Name: value, dtype: float64

I saved it into Postgres database using:

df.to_sql('test_pp', con=engine)

After when I retrieve it back from the DB as a new_df: (edit: actual df)

date
2019-11-01 07:30:00+00:00    1148.144708
2019-11-01 07:45:00+00:00    1148.015876
2019-11-01 08:00:00+00:00    1147.911068
2019-11-01 08:15:00+00:00    1147.925240
2019-11-01 08:30:00+00:00    1148.007117
2019-11-01 08:45:00+00:00    1148.122853
2019-11-01 09:00:00+00:00    1148.251176
2019-11-01 09:15:00+00:00    1148.362736
2019-11-01 09:30:00+00:00    1148.423236
2019-11-01 09:45:00+00:00    1148.528360
Name: value, dtype: float64

both these dataframes are not equal. Why is that?

df.equals(new_df)

returns False. Any idea? P.S.: I also checked the order and shape and sorted it based on the date.

I also checked the indexes

DatetimeIndex(['2019-11-01 09:45:00+00:00'], dtype='datetime64[ns, UTC]', name='date', freq=None),
DatetimeIndex(['2019-11-01 09:45:00+00:00'], dtype='datetime64[ns, UTC]', name='date', freq=None))

and the dtypes

df['value'].dtypes, new_df['value'].dtypes

returns

(dtype('float64'), dtype('float64'))

strangely,

df['value'].iloc[-1:].index == new_df['value'].iloc[-1:].index

returns True

and

df['value'].iloc[-1].values == new_df['value'].iloc[-1].values

returns False

Upvotes: 1

Views: 77

Answers (1)

stargazer
stargazer

Reputation: 33

It's because of the double precision in Postgres. It rounded off the last 3 digits.

Upvotes: 1

Related Questions