CODEWITHSUNDEEP
scaladatetimeapache-sparknullunix-timestamp
Mat.cort
Mat.cort

Reputation: 37

Problems to convert date to timestamp, Spark date to timestamp from unix_timestamp return null

Problems to convert date to timestamp, Spark date to timestamp from unix_timestamp return null.

scala>  import org.apache.spark.sql.functions.unix_timestamp
scala> spark.sql("select from_unixtime(unix_timestamp(('2017-08-13 00:06:05'),'yyyy-MM-dd HH:mm:ss')) AS date").show(false)

+----+
|date|
+----+
|null|
+----+

Upvotes: 2

Views: 1483

Answers (1)

Mat.cort
Mat.cort

Reputation: 37

The problem was the change of time in Chile, thank you very much.

+-------------------+---------+
|     DateIntermedia|TimeStamp|
+-------------------+---------+
|13-08-2017 00:01:07|     null|
|13-08-2017 00:10:33|     null|
|14-08-2016 00:28:42|     null|
|13-08-2017 00:04:43|     null|
|13-08-2017 00:33:51|     null|
|14-08-2016 00:28:08|     null|
|14-08-2016 00:15:34|     null|
|14-08-2016 00:21:04|     null|
|13-08-2017 00:34:13|     null|
+-------------------+---------+

The solution, set timeZone:

  spark.conf.set("spark.sql.session.timeZone", "UTC-6")

Upvotes: 2

Related Questions