CODEWITHSUNDEEP
rapache-sparksparkr
sag
sag

Reputation: 5451

Change dataType of spark dataframe column in SparkR

I have a spark dataframe. 

root
 |-- col1: string (nullable = true)
 |-- Id: string (nullable = true)
 |-- col2: string (nullable = true)
 |-- col3: string (nullable = true)
 |-- col4: string (nullable = true)
 |-- date1: string (nullable = true)
 |-- col5: string (nullable = true)
 |-- date2: string (nullable = true)

I just want to convert the date2 column to date. I used the below code to do that

to_date(myDF$date2)

But the dataframe remains the same. No change in date2 datatype.

How can I change the column date2 to date dataType?

Upvotes: 2

Views: 3963

Answers (1)

Michael Neylon
Michael Neylon

Reputation: 624

myDF <- withColumn(myDF, "date2", cast(myDF$date2, "date"))

See these pages in the SparkR documentation for the latest version (2.0.1 at the time of writing this):

The cast will not change the existing dataframe, so you need to create a new dataframe or replace the existing dataframe with the newly casted column replacing the old column of the same name.

Upvotes: 4

Related Questions