CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
Learn Hadoop
Learn Hadoop

Reputation: 3060

Spark SQL between statement

How to find the salary within a range? Here is my Spark SQL dataframe code for finding the salary between 10000 and 20000:

empData.where($"Salary" > 10000 && $"Salary" < 20000).orderBy($"Salary".desc).show()

I believe there should be an alternative solution using between. How can I use between approach?

Upvotes: 1

Views: 33416

Answers (3)

Davoud Malekahmadi
Davoud Malekahmadi

Reputation: 63

empData.filter(col("Salary").between(10000 ,20000)).orderBy(col("Salary").desc());

or

empData.where(col("Salary").between(10000 ,20000)).orderBy(col("Salary").desc());

Upvotes: 0

Navom Saxena
Navom Saxena

Reputation: 21

If you wish to use between, you can use sparkSQL and run logic as query. For e.g.

empData.createOrReplaceTempView("empDataTempTable")

val filteredData = spark.sql("select * from empDataTempTable where salary between 10000 and 20000 order by salary desc")

 filteredData.show()

For random lookups in a column and filter process, sparkSQL and DataFrame gives almost same performance results.

Upvotes: 2

user10494677
user10494677

Reputation:

I believe have alternative solution to use between.

Just like that:

empData.where($"Salary".between(10000, 20000))

Upvotes: 12

Related Questions