CODEWITHSUNDEEP
rapache-sparkapache-spark-sqlsparkr
user124543131234523
user124543131234523

Reputation: 283

SparkR - cast to date format

How do I cast string to date with a specific format for a Spark dataframe?

In dplyr, I would do this:

df = data.frame(dt1 = c("22DEC16", "12JUN16"), x = c(10,20))
df = df %>% mutate(dt2 = as.Date(dt1, "%d%b%y"))

> df
  dt1     x        dt2
1 22DEC16 10 2016-12-22
2 12JUN16 20 2016-06-12

Upvotes: 0

Views: 1924

Answers (1)

zero323
zero323

Reputation: 330343

In Spark 2.2 or later:

library(magrittr)

df <- createDataFrame(data.frame(dt=c("22DEC16", "12JUN16")))

df %>% withColumn("parsed", to_date(.$dt,  "ddMMMyy")) %>% head()

       dt     parsed
1 22DEC16 2016-12-22
2 12JUN16 2016-06-12

Before Spark 2.2:

df %>%
  withColumn(
   "parsed",
   unix_timestamp(.$dt, "ddMMMyy") %>% cast("timestamp") %>% cast("date")) %>%
  head()

       dt     parsed
1 22DEC16 2016-12-22
2 12JUN16 2016-06-12

With this solution you should watch for possible timezone inconsistencies.

Upvotes: 3

Related Questions