CODEWITHSUNDEEP
pyspark
Alejandro Fleitas
Alejandro Fleitas

Reputation: 1

Pyspark problem when covert string to datetime

I have problem when I trying to convert string to datetime in pyspark, my problem is that the result is always null.

enter image description here

Upvotes: 0

Views: 204

Answers (1)

Vaebhav
Vaebhav

Reputation: 5032

You need to provide the date_format to to_date to parse the dates -

Data Preparation

sparkDF = sql.createDataFrame(
        [
          (12312,"06/02/2020"),
          (124567,"05/04/2020"),
          (123124,"27/09/2021"),
          (124214,"31/01/2022"),
        ],
        ("cust", "start_date")
    )

sparkDF.show()

+------+----------+
|  cust|start_date|
+------+----------+
| 12312|06/02/2020|
|124567|05/04/2020|
|123124|27/09/2021|
|124214|31/01/2022|
+------+----------+

To Date

sparkDF = sparkDF.withColumn('start_date_2',F.to_date(F.col('start_date'), 'dd/MM/yyyy'))

sparkDF.show()

+------+----------+------------+
|  cust|start_date|start_date_2|
+------+----------+------------+
| 12312|06/02/2020|  2020-02-06|
|124567|05/04/2020|  2020-04-05|
|123124|27/09/2021|  2021-09-27|
|124214|31/01/2022|  2022-01-31|
+------+----------+------------+

Upvotes: 1

Related Questions