CODEWITHSUNDEEP
apache-sparkdataframe
MetallicPriest
MetallicPriest

Reputation: 30745

How to convert String to spark.sql.Column for querying?

The situation is that I want to give the user the ability to filter out data from a dataframe. Progammatically I can do it fine, like this.

val filter = col("SomeColA") > 0.1 && col("SomeColB") > 0.2

val queriedData = sqlContext.read.format("csv").
 option("header", "true").
 option("delimiter", "\t").option("inferSchema","true").
 load(filePath).where(filter)

However, here I want the user to provide the filter itself, using for example a text box. In other words, I want to be able to turn a string into its equivlant spark.sql.Column value. For example, I want the user to be able to specify $"SomeCol" > 0.1 here in the form of a String. How can I do that?

Upvotes: 0

Views: 919

Answers (1)

ollik1
ollik1

Reputation: 4540

One can simply use the version of where that takes the condition expression as a string i.e. .where("SomeCol > 0.1"). See https://spark.apache.org/docs/2.4.0/api/java/org/apache/spark/sql/Dataset.html#where-java.lang.String-

Upvotes: 2

Related Questions