CODEWITHSUNDEEP
amazon-web-servicesamazon-s3parquetaws-glueamazon-athena
SK Sayyad
SK Sayyad

Reputation: 85

How do I convert String to date in AWS Glue?

When I run crawler from Glue on parquet/csv file in S3 bucket, It takes Date as a string. I changed in edit schema and set it to Date. Not only date but it's not changing any data type in edit schema. When I fire query from Athena "select order_date from parque_sales", the error is:

HIVE_BAD_DATA: Field order_date's type BINARY in parquet is incompatible with type date defined in table schema

I also tried changing schema in Glue studio, the result is same!

Thanks in advance! Please help!

Upvotes: 1

Views: 2399

Answers (1)

Nishu Tayal
Nishu Tayal

Reputation: 20840

It seems that you have files where date is stored as string in parquet, not as date. The crawler still receives those as string values and fails while comparing the source schema with the defined schema.

To avoid this issue, you should use explicit casting in the athena query. Something like this :

select date_parse(datestring,'%Y-%m-%d %h:%i:%s')

Upvotes: 1

Related Questions