CODEWITHSUNDEEP
python
Piotr Z
Piotr Z

Reputation: 23

Convert String to Date column

I want to convert String column to Date column, but as a result I receive column with null values.

from pyspark.sql.functions import expr, from_unixtime, dayofmonth, unix_timestamp, year, to_date, col

dane3.withColumn('date', to_date(unix_timestamp(col('dateRep'),'%d.%m.%Y').cast("timestamp"))).show()

enter image description here

Upvotes: 1

Views: 81

Answers (1)

Equinox
Equinox

Reputation: 6748

You don't need the % before d/m/y like sql. More info in the docs

df = spark.createDataFrame(
    [
        (1, '27.08.2020'), 
        (2, '27.08.2019'),
    ],
    ['id', 'txt']
)

df = df.withColumn('formatted', 
                   to_date(unix_timestamp(col('txt'), 'dd.MM.yyyy').cast("timestamp")))
df.show()

+---+----------+----------+
| id|       txt| formatted|
+---+----------+----------+
|  1|27.08.2020|2019-12-29|
|  2|27.08.2019|2018-12-30|
+---+----------+----------+

Upvotes: 1

Related Questions