CODEWITHSUNDEEP
apache-sparkapache-spark-sql
Sudheer Nulu
Sudheer Nulu

Reputation: 55

Spark SQL function to count the repetation

Trying to achieve the below task using spark sql......

I have a table values like

table structure

I need the output like

1               1 has repeated for 3,3 has repeated for 3 times.....
2               2 has repeated for 3,3 has repeated for two times.......

from my end I have grouped the data using row1 to get same id's together

scala> val data=rows.groupBy("row1")

after that i am not able to split the row2 with "," and count the occurrence of each value...

Could any one help on this..

Upvotes: 0

Views: 81

Answers (1)

user8783971
user8783971

Reputation: 1

Split and explode:

import org.apache.spark.sql.functions._

df.select(col("row1"), explode(split(col("row2"), ",")).alias("row2"))
  .groupBy("row1", "row2").count

Upvotes: 1

Related Questions