CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
Vicky C
Vicky C

Reputation: 53

Filtering out data in Spark dataframe in Scala

I have a dataframe df, which contains below data:

**customers**   **product**   **Val_id**
     1               A            1
     2               B            X
     3               C               
     4               D            Z

I have successfully filtered for data where column val_id is blank

df.where(col("val_id").isin(""))

But I am not able to figure out a way to filter data where column val_id is not blank, i tried something like below, but did not work for me: 

df.where(col("val_id").isnotin(""))

Can anyone please help me to achieve it using Spark Scala.

Upvotes: 1

Views: 9579

Answers (3)

Varun Chadha
Varun Chadha

Reputation: 376

You can use filter to get desired output:

df.filter("rule_id != ''")

Upvotes: 3

Leo C
Leo C

Reputation: 22439

Assuming Val_id is of String type, you can use this inequality operator !==:

df.where(col("Val_id") !== "").show

Conversely, you can also use === for matching the blank.

Upvotes: 2

user8107573
user8107573

Reputation: 1

If column type is String:

df.where(trim(col("val_id")) != "")

Upvotes: 0

Related Questions