CODEWITHSUNDEEP
apache-spark-sqlpyspark
hellotherebj
hellotherebj

Reputation: 121

how to change a string to Date format?

This might be a duplicate somewhere, but I have a simple df:

df1_schema = StructType([StructField("Date", StringType(), True) ])
df_data = [('1-Jun-20',)]
rdd = sc.parallelize(df_data)
df1 = sqlContext.createDataFrame(df_data, df1_schema)
#df1 = df1.withColumn("Date",to_date("Date", 'yyyy-MM-dd'))
df1.show()

+--------+
|    Date|
+--------+
|1-Jun-20|
+--------+

I tried to change to Date format but it just gives me a null value. This is what I tried:

df1= df1.withColumn("Date2", F.to_date(col('Date'), "dd-MM-yyyy"))
+----------+-----+
|      Date|Date2|
+----------+-----+
|1-Jun-20  | null|
+----------+-----+

any solution to this?.. Thank you

Upvotes: 0

Views: 118

Answers (1)

Cena
Cena

Reputation: 3419

The correct format for your Date is "d-MMM-yy"

df1.withColumn("Date2", F.to_date(col('Date'), "d-MMM-yy")).show()
+--------+----------+
|    Date|     Date2|
+--------+----------+
|1-Jun-20|2020-06-01|
+--------+----------+

This also works for 01-Jun-20 or 10-Jun-20.

Upvotes: 1

Related Questions