CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
Manoj Kumar
Manoj Kumar

Reputation: 29

Extracting Year,Month And Hour from a column using Spark Scala

I have a data file in which a column datetime has value 01-01-2011 00:00. I want to extract year, month and hour from this value. Code:

val actualData=df1.select(year(col("datetime")).as("Year"),
month(col("datetime")).as("Month"),
hour(col("datetime")).as("Hour"),
dayofmonth(col("datetime")).as("DayOfMonth"))

Output i am getting:

+---------------------+----+-----+----+----------+
Year|Month|Hour|DayOfMonth|
+---------------------+----+-----+----+----------+
|null| null|null| null|
|null| null|null| null|
|null| null|null| null|
|null| null|null| null|

Upvotes: 0

Views: 1081

Answers (2)

shengkui
shengkui

Reputation: 21

The format of "datetime" is not "yyyy-MM-dd HH:mm", so it need to specify the format of it.

val actualData=df1.select(year(to_date(col("datetime"), "dd-MM-yyyy HH:mm")).as("Year"),
month(to_date(col("datetime"), "dd-MM-yyyy HH:mm")).as("Month"),
hour(to_timestamp(col("datetime"), "dd-MM-yyyy HH:mm")).as("Hour"),
dayofmonth(to_date(col("datetime"), "dd-MM-yyyy HH:mm")).as("DayOfMonth"))

Upvotes: 1

Vikram Patel
Vikram Patel

Reputation: 79

Seems column datetime is not in appropriate datatype. Casting the datetime column to date should help.

    df1 = df1.withColumn("datetime",to_date(col("datetime"),"dd-MM-yyyy HH:mm"))

Upvotes: 2

Related Questions