CODEWITHSUNDEEP
excelcsvapache-sparkpysparktimestamp
Ambarish Srinivasan
Ambarish Srinivasan

Reputation: 9

Apache Spark time only column issue

I am trying to read a csv and excel using Apache Spark with inferschema = True.

For CSV I am using Spark Core Read API Function (spark.read.format('csv')).

For Excel I am using spark-excel jar (spark.read.format('excel')).

In Both CSV & Excel File, I am having a time only column (HH:MM:SS) (Ex: 12:18:34)

When I tried to read csv, I am getting the df with timestamp column of data like (2025-02-19 12:18:34)

In excel read, I am getting the df like (1899-12-31 12:18:34)

Why this issue happens and is there a way to overcome this while having the inferschema = True option.

Upvotes: -1

Views: 59

Answers (0)

Related Questions