CODEWITHSUNDEEP
apache-sparkpysparkapache-spark-sql
Susanth
Susanth

Reputation: 21

How to convert String to Time in PYSPARK?

iam trying to convert string to Time, but getting NULL.

For Ex, val Start =='080000'

used the below steps,

1)unix_timestamp(col('Start'),'HH:mm:ss'),\
2)to_timestamp(lit('Start'),'HH:mm:ss'),\
3)to_timestamp(col('Start'),'HH:mm:ss'),\
4)from_unixtime(unix_timestamp(col('Start'),'HH:mm:ss'))

Expected Output :

08:00:00 (HH:MM:SS)

could someone please suggest the methods

Upvotes: 0

Views: 1290

Answers (1)

pltc
pltc

Reputation: 6082

Spark does have TimeType. Latest version v3.1.1 only has DateType and TimestampType, so the simple answer to your request converting String to Time is impossible.

However, it's possible to convert from 080000 (StringType) to 2000-01-01 08:00:00 (TimestampType) – or any date as the date doesn't matter – and you can perform any kind of date comparison you want

(df
    .withColumn('from_timestamp', F.regexp_replace(F.col('from'), '(\d{2})(\d{2})(\d{2})', '2000-01-01 $1:$2:$3'))
    .withColumn('to_timestamp', F.regexp_replace(F.col('to'), '(\d{2})(\d{2})(\d{2})', '2000-01-01 $1:$2:$3'))
    .withColumn('diff', F.to_timestamp(F.col('to_timestamp')) - F.to_timestamp(F.col('from_timestamp')))
    .show()
)

# +------+------+-------------------+-------------------+----------+
# |  from|    to|     from_timestamp|       to_timestamp|      diff|
# +------+------+-------------------+-------------------+----------+
# |080000|083000|2000-01-01 08:00:00|2000-01-01 08:30:00|30 minutes|
# +------+------+-------------------+-------------------+----------+

Upvotes: 1

Related Questions