CODEWITHSUNDEEP
apache-sparkpyspark
f.ivy
f.ivy

Reputation: 75

Converting Double type column to date format type pyspark retuning

I have a dataframe with double column (20180310), i want to convert it to Date format (2018-03-10).

I used this function:

df.withColumn(
    "new_col", 
    F.to_date(F.col("old_col").cast("string"), "yyyyMMdd")
)

but it's returning error message or Null.

Do you have any idea please?

Upvotes: 1

Views: 1795

Answers (1)

Luiz Viola
Luiz Viola

Reputation: 2436

Because when you cast from double to string, the column will have this form: 2.018031E7

cast to integer before casting to string:

df.withColumn( "new_col", F.to_date(F.col("old_col").cast("integer").cast("string"), "yyyyMMdd"))

Upvotes: 2

Related Questions