CODEWITHSUNDEEP
pythonapache-sparkpysparkapache-spark-sql
user9548476
user9548476

Reputation:

Pyspark date format

I need to convert a string into a date format, for this the string format I have is something like this:

2021-01-07 11:17:49.385820+00:00

and what I am doing is the following:

    format = "yyyy-MM-dd HH:mm:ss.SSSSSS+ZZ:ZZ"
    level_1_data_df = level_1_data_value_df\
        .select(
        "level_1_data.*"
        ).withColumn("time2", to_timestamp(col("time"), format))

but the result of the new field is null, any ideas?

Upvotes: 1

Views: 90

Answers (1)

mck
mck

Reputation: 42422

You can simply do

.withColumn("time2", to_timestamp(col("time")))

or, equivalently,

.withColumn("time2", col("time").cast("timestamp"))

because your timestamp has a standard format. No need for specifying its format.

Upvotes: 1

Related Questions