CODEWITHSUNDEEP
scalaapache-sparkspark-streaming
Jean
Jean

Reputation: 429

filter the lines by two words Spark Streaming

Is there a way to filter with one expression the lines containing a word "word1" or the other "word2" something like :

val res = lines.filter(line => line.contains("word1" or "word2"))

because this expression doesn't work.

Thank you in advance

Upvotes: 2

Views: 1339

Answers (1)

zero323
zero323

Reputation: 330323

If line is a String optimal choice would regexp:

val pattern = "word1|word2".r

lines.filter(line => pattern.findFirstIn(line).isDefined)

otherwise (other sequence type) you can use Seq.exists:

lines.filter(line => Seq("foo", "bar").exists(s => line.contains(s)))

which takes a single which maps from element to boolean (here (String) ⇒ Boolean) and:

tests whether a predicate holds for at least one element of this iterable collection.

Upvotes: 5

Related Questions