CODEWITHSUNDEEP
pythondatetimepysparkapache-spark-sqltimestamp
Shankar
Shankar

Reputation: 27

How to convert date string to timestamp format in pyspark

I need to convert string '07 Dec 2021 04:35:05' to date format 2021-12-07 04:35:05 in pyspark using dataframe or spark sql.

Upvotes: 0

Views: 6306

Answers (1)

Ric S
Ric S

Reputation: 9277

If you have a column full of dates with that format, you can use to_timestamp() and specify the format according to these datetime patterns.

import pyspark.sql.functions as F

df.withColumn('new_column', F.to_timestamp('my_column', format='dd MMM yyyy HH:mm:ss'))

Example

import pyspark.sql.functions as F

df = spark.createDataFrame([
  ('07 Dec 2021 04:35:05', ),
  ('31 Dec 2021 06:45:15', )
], ('my_column', ))

df.withColumn('new_column', F.to_timestamp('my_column', format='dd MMM yyyy HH:mm:ss')) \
    .show()
+--------------------+-------------------+
|           my_column|         new_column|
+--------------------+-------------------+
|07 Dec 2021 04:35:05|2021-12-07 04:35:05|
|31 Dec 2021 06:45:15|2021-12-31 06:45:15|
+--------------------+-------------------+

Upvotes: 2

Related Questions