CODEWITHSUNDEEP
rdatetimeapache-sparkapache-spark-sqlsparkr
AltShift
AltShift

Reputation: 346

Convert string to datetime in SparkR?

I have a Formal Class DataFrame object that was uploaded to SparkR from MySQL (via a json file), which contains formatted strings like this: "2012-07-02 20:14:00"

I need to convert these to a datetime type in SparkR, but this does not seem to be supported yet. Is there an undocumented function or a recipe for doing this with a UDF? (Nb. I haven't actually tried creating a SparkR UDF before, so I'm grasping at straws, here.)

Upvotes: 2

Views: 2432

Answers (1)

zero323
zero323

Reputation: 330343

Spark SQL doesn't support R UDFs but in this particular case you can simply cast to timestamp: 

df <- createDataFrame(sqlContext, 
  data.frame(dts=c("2012-07-02 20:14:00", "2015-12-28 00:10:00")))
dfWithTimestamp <- withColumn(df, "ts", cast(df$dts, "timestamp"))

printSchema(dfWithTimestamp)
## root
##  |-- dts: string (nullable = true)
##  |-- ts: timestamp (nullable = true)

head(dfWithTimestamp)
##                   dts                  ts
## 1 2012-07-02 20:14:00 2012-07-02 20:14:00
## 2 2015-12-28 00:10:00 2015-12-28 00:10:00

Upvotes: 5

Related Questions