CODEWITHSUNDEEP
scaladateapache-sparkdataframe
Khaoula Arfaoui
Khaoula Arfaoui

Reputation: 75

How to convert string into a date format in spark

I have passed a string (datestr) to a function (that do ETL on a dataframe in spark using scala API) however at some point I need to filter the dataframe by a certain date something like : 

df.filter(col("dt_adpublished_simple") === date_add(datestr, -8))

where datestr is the parameter that I passed to the function.

Unfortunately, the function date_add requires a column type as a first param. Can anyone help me with how to convert the param into a column or a similar solution that will solve the issue?

Upvotes: 2

Views: 184

Answers (1)

Luis Miguel Mejía Suárez
Luis Miguel Mejía Suárez

Reputation: 22895

You probably only need to use lit to create a String Column from your input String. And then, use to_date to create a Date Column from the previous one.

df.filter(col("dt_adpublished_simple") === date_add(to_date(lit(datestr), format), -8))

Upvotes: 2

Related Questions