CODEWITHSUNDEEP
pandasdatetime64ibis
Brian Chapman
Brian Chapman

Reputation: 11

Specifying datetime64 resolution in Ibis when converting to Pandas DataFrame

I have a MySQL database with datetime values shifted by arbitrary amounts for de-identification purposes. So, for example, I have a date value of datetime.datetime(2644, 1, 17, 0, 0) . If I query these values with pymysql or Pandas I get a fine datetime object. However, if I use Ibis to construct the query I get a failure because the dates fall outside of the data range that can be represented with the datetime64[ns] data type that Ibis is using in the conversion to the DataFrame.

So

---------------------------------------------------------------------------
OutOfBoundsDatetime                       Traceback (most recent call last)
~/opt/anaconda3/envs/clinicalnlp/lib/python3.7/site-packages/pandas/core/arrays/datetimes.py in objects_to_datetime64ns(data, dayfirst, yearfirst, utc, errors, require_iso8601, allow_object)
   2084         try:
-> 2085             values, tz_parsed = conversion.datetime_to_datetime64(data)
   2086             # If tzaware, these values represent unix timestamps, so we

pandas/_libs/tslibs/conversion.pyx in pandas._libs.tslibs.conversion.datetime_to_datetime64()

pandas/_libs/tslibs/np_datetime.pyx in pandas._libs.tslibs.np_datetime.check_dts_bounds()

OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 2644-01-17 06:48:00

Browsing through the Ibis source code, it seems like it should be possible to configure the datetime64 time unit from nanoseconds to microseconds, so that the dates fall within the allowable range, but I cannot figure out how to do this configuration within Ibis.

Any suggestions would be greatly appreciated.

Upvotes: 1

Views: 332

Answers (0)

Related Questions