CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sqlspark-streaming
JDev
JDev

Reputation: 1822

spark scala how can I calculate days since 1970-01-01

Looking for scala code to replicate https://www.epochconverter.com/seconds-days-since-y0

I have a spark streaming job reading the avro message. The message has a column of type int and holds Days Since 1970-01-01. I want to convert that to date.

 dataFrame.select(from_avro(col("Value"), valueRegistryConfig) as 'value)
 .select("value.*")
 .withColumn("start_date",'start_date)

start_date is holding an integer value like 18022 i.e Days Since 1970-01-01. I want to convert this value to a date

18022 - > Sun May 05 2019

Upvotes: 0

Views: 870

Answers (1)

Ramdev Sharma
Ramdev Sharma

Reputation: 1014

Use default date as 1970-01-01 and pass number of days to date_add function.

This will give you date but will be 1 day additional so you do minus 1.

Something like this:

var dataDF = Seq(("1970-01-01",18091),("1970-01-01",18021),("1970-01-01",18022)).toDF("date","num")
dataDF.select(
    col("date"),
    expr("date_add(date,num-1)").as("date_add")).show(10,false)

+----------+----------+
|date      |date_add  |
+----------+----------+
|1970-01-01|2019-07-13|
|1970-01-01|2019-05-04|
|1970-01-01|2019-05-05|
+----------+----------+

Upvotes: 2

Related Questions