CODEWITHSUNDEEP
scalaapache-sparkk-means
Royce
Royce

Reputation: 1595

How to get min and max of clusters

I created a scala program to apply k-means on a specific column of a dataframe. Dataframe name is df_items and column name is price.

import org.apache.spark._
import org.apache.spark.sql.types._
import org.apache.spark.ml.clustering._
import org.apache.spark.ml.feature.VectorAssembler 

val df_items  = spark.read.format("csv").option("header","true").load(path.csv)

// need to cast because df_items("price") is String
df_items.createGlobalTempView("items")
val price = spark.sql("SELECT cast(price as double) price FROM global_temp.items")
case class Rows(price:Double)
val rows = price.as[Rows]
val assembler = new VectorAssembler().setInputCols(Array("price")).setOutputCol("features")
val data = assembler.transform(rows)
val kmeans = new KMeans().setK(6)
val model = kmeans.fit(data)
val predictions = model.summary.predictions

Predictions result :

+------+--------+----------+
| price|features|prediction|
+------+--------+----------+
|  58.9|  [58.9]|         0|
| 239.9| [239.9]|         3|
| 199.0| [199.0]|         5|
| 12.99| [12.99]|         0|
| 199.9| [199.9]|         5|
|  21.9|  [21.9]|         0|
|  19.9|  [19.9]|         0|
| 810.0| [810.0]|         1|
|145.95|[145.95]|         5|
| ...  |   ...  |    ...   |

My goal is to get the min and the max value of a cluster (or all clusters). It is possible?

Thank's a lot

Upvotes: 0

Views: 903

Answers (1)

astro_asz
astro_asz

Reputation: 2318

If I understand your question correctly, you could use groupBy to group by prediction column.

predictions.groupBy("prediction")
           .agg(min(col("price")).as("min_price"),
                max(col("price")).as("max_price"))

Is this what you need?

Upvotes: 3

Related Questions