CODEWITHSUNDEEP
sqldataframedatepysparkdatabricks
shreder1921
shreder1921

Reputation: 101

how to convert a column with dates(string) to a date only in pyspark

i have a dataframe with a "created_at" column that contains dates that look like this:

Mon Jan 27 17:16:46 +0000 2020

and i want to convert this column dates to:

2020-01-27

how can i do this in pyspark?

Upvotes: 2

Views: 52

Answers (1)

murtihash
murtihash

Reputation: 8410

Refer to Jave SimpleDate Format for more details regarding datetime characters.

df.show(truncate=False) #sample dataframe

#+------------------------------+
#|created_at                    |
#+------------------------------+
#|Mon Jan 27 17:16:46 +0000 2020|
#+------------------------------+

from pyspark.sql import functions as F

df.withColumn("created_at", F.to_date("created_at","EEE MMM dd HH:mm:ss +SSSS yyyy")).show()

#+----------+
#|created_at|
#+----------+
#|2020-01-27|
#+----------+

Upvotes: 3

Related Questions