CODEWITHSUNDEEP
scalaapache-sparkdataframe
user9486876
user9486876

Reputation:

Check that a SPARK Dataframe column matches a Regex for all occurrences using Scala

I am using Scala.

I have a dataframe with a column date which looks like that:

|            date             |
|2017-09-24T11:05:52.647+02:00|
|2018-09-24T11:05:52.647+02:00|
|2018-10-24T11:05:52.647+02:00|

I have a regex to check the date format:

pattern = new regex(([12]\d{3}-(0[1-9]|1[0-2])-(0[1-9]|    [12]\d|3[01])T\d{2}:\d{2}:\d{2}.\d{3}\+\d{2}:\d{2}))

I want to check if each row in the dataframe matches with the regex, if yes return true and if not return false. I need to return just true or false not a list.

Any help is welcome and many thanks for your help.

Upvotes: 1

Views: 3733

Answers (1)

Ged
Ged

Reputation: 18003

This should work - but turning it around, find first non-match: 

import scala.util.Try
val result = Try(Option(df.filter($"cityid" rlike "[^0-9]").first)).toOption.flatten
if (result.isEmpty) { println("Empty")}

I use a DF as outcome and you can just check if empty or not.

Please tailor to your own situation. e.g. not empty, your own regex.

Without the Try and such the .first generates an error if empty. None is returned if empty and you can do the empty check.

Upvotes: 1

Related Questions