CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sqlspark-streamingapache-spark-2.2
stackoverflow
stackoverflow

Reputation: 69

how to get the row corresponding to the minimum value of some column in spark scala dataframe

i have the following code. df3 is created using the following code.i want to get the minimum value of distance_n and also the entire row containing that minimum value .

enter image description here

 //it give just the min value , but i want entire row containing that min value

enter image description here

for getting the entire row , i converted this df3 to table for performing spark.sql

if i do like this spark.sql("select latitude,longitude,speed,min(distance_n) from table1").show()

//it throws error enter image description here

and if spark.sql("select latitude,longitude,speed,min(distance_nd) from table180").show()

// by replacing the distance_n with distance_nd it throw the error

enter image description here

how to resolve this to get the entire row corresponding to min value

Upvotes: 2

Views: 1340

Answers (1)

Constantine
Constantine

Reputation: 1416

Before using a custom UDF, you have to register it in spark's sql Context.

e.g:

spark.sqlContext.udf.register("strLen", (s: String) => s.length())

After the UDF is registered, you can access it in your spark sql like

spark.sql("select strLen(some_col) from some_table")

Reference: https://docs.databricks.com/spark/latest/spark-sql/udf-scala.html

Upvotes: 1

Related Questions