CODEWITHSUNDEEP
pythonpandasamazon-athenapyarrow
kismsu
kismsu

Reputation: 1099

Best way to save pandas DataFrame to parquet with date type

I'm trying to save DataFrame with date type column to a parquet format to be used later in Athena. As far as I understand parquet has native DATE type, by the only type I can really use is datetime64[ns] with pyarrow engine (here is the same issue discussed https://github.com/pandas-dev/pandas/issues/20089). The issue is I'd like to have date type rather than datetime in Athena schema. Any suggestions?

Upvotes: 3

Views: 10088

Answers (1)

Micah Kornfield
Micah Kornfield

Reputation: 1718

As mentioned in the comment I believe Apache Arrow 0.15.1 now supports round-tripping dates between Pandas and Parquet.

Upvotes: 3

Related Questions