CODEWITHSUNDEEP
datetimepyspark
Chelseajcole
Chelseajcole

Reputation: 547

PySpark Convert String Column to Datetime Type

I have the TIMESTAMP data like:

[29:23:59:45]

This stands for whatever month 29, 23:59:45

How can I convert in PySpark to like DAY 29, TIME:23:59:45?

Possibly using something like

from datetime import datetime

dVal = datetime.strptime('[29:23:59:45]', '%d/%h/%m/%s')

Upvotes: 0

Views: 86

Answers (1)

Gian Pio Domiziani
Gian Pio Domiziani

Reputation: 129

This is a classic example for which is needed to use a User Defined Function (UDF).

from datetime import datetime
from spark.sql import functions as F

def toDate(x):
   return datetime.strptime(x, '%m %H:%M:%S')

toDate = F.udf(toDate)
new_df = df.withColumn('date', toDate(F.col('timestamp'))

where, df is supposed to be the old dataframe containing a column named 'timestamp' as you reported.

Upvotes: 1

Related Questions