CODEWITHSUNDEEP
pythonapache-sparkpysparkapache-spark-sql
Farid Arshad
Farid Arshad

Reputation: 334

pyspark string to date

I am trying to convert the string to date format, Date column consist data in such order but this are in string datatype

20191130
20191231

when using string to date, date should display as

2019-11-31
2019-12-31

  1. I tried this approach but script returned error

     df = spark.sql('select * from tablename) 
 df2 = df.withColumn('Date', expr("cast(as_of_date,'yyyyMMdd) as date"))

  2. I also tried on this script and it works , however, with this , it is displaying date and time which is not I wanted

    df2 = df.withColumn("Date",expr("cast(unix_timestamp(as_of_date ,'yyyyMMdd') as date)")).show()

Upvotes: 0

Views: 62

Answers (1)

mck
mck

Reputation: 42422

Try using to_date?

df2 = df.withColumn('Date', to_date(col('as_of_date'), 'yyyyMMdd'))

Upvotes: 1

Related Questions