CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
gstvolvr
gstvolvr

Reputation: 650

Spark SQL filter multiple fields

What is the corrent syntax for filtering on multiple columns in the Scala API? If I want to do something like this:

dataFrame.filter($"col01" === "something" && $"col02" === "something else")

or

dataFrame.filter($"col01" === "something" || $"col02" === "something else")

EDIT:

This is what my original code looks like. Everything comes in as a string. 

df.select($"userID" as "user", $"itemID" as "item", $"quantity" cast("int"), $"price" cast("float"), $"discount" cast ("float"), sqlf.substring($"datetime", 0, 10) as "date", $"group")
  .filter($"item" !== "" && $"group" !== "-1")

Upvotes: 16

Views: 60445

Answers (1)

dheee
dheee

Reputation: 1658

I think i see what the issue is. For some reason, spark does not allow two !='s in the same filter. Need to look at how filter is defined in Spark source code.

Now for your code to work, you can use this to do the filter

df.filter(col("item").notEqual("") && col("group").notEqual("-1"))

or use two filters in same statement

df.filter($"item" !== "").filter($"group" !== "-1").select(....)

This link here can help with different spark methods.

Upvotes: 23

Related Questions